8

Findings and Recommendations

The findings and recommendations in this chapter reflect the performance of the DoE Small Business Innovation Research (SBIR) and Small Business Technology Transfer (STTR) program against the broad congressional objectives for the programs.¹

For SBIR, these objectives were reiterated in the 2011 program reauthorization and elaborated in the subsequent policy directive of the Small Business Administration (SBA).² Section 1c of the SBA SBIR Directive states program goals as follows:

___________________

¹ See Box 1-3 and the discussion of the committee’s task in Chapter 1 (Introduction).

² SBA SBIR-STTR Policy Directive, October 18, 2012.

³ Small Business Administration, Small Business Innovation Research (SBIR) Program Policy Directive, February 24, 2014.

The parallel language from the SBA’s STTR Policy Directive is as follows:

This chapter reviews the extent to which each of these program goals is being addressed at DoE. The chapter also addresses some specific aspects of DoE’s management of the program.

SOURCES OF FINDINGS

The committee’s findings are based on a complement of quantitative and qualitative tools including a survey, case studies of award recipients, agency data, public workshops, and agency meetings. The methodology is described in Chapter 1 and Appendix A of this report. In reviewing the findings below, it is important to note that the National Academies of Sciences, Engineering, and Medicine 2014 Survey—hereafter referred to as the 2014 Survey—was sent to every principal investigator (PI) who won a Phase II award from DoE, FY 2001-2010 (not the registered company points of contact [POC] for each company.)⁵ Each PI was asked to complete a maximum of two questionnaires, which as a result excludes some awards from the survey. The preliminary population was developed by taking the original set of SBIR and STTR Phase II awards made by DoE during the study period and eliminating on a random basis awards to PIs who received more than two awards (to limit the burden on respondents). The resulting preliminary population was 1,077 awards. PIs for 583 of these awards were determined to be not contactable at the SBIR/STTR company listed in the DoE awards database. The remaining 494 awards constitute the effective population for this study. From the effective population, we received 269 responses. As a result, the response rate in relation to the preliminary population was 25.0 percent and in relation to the effective population was 54.5 percent.

In addition to information from this survey, the committee has drawn on company case studies, discussions with agency staff, and other documentation. In

___________________

⁴ Small Business Administration, Office of Investment and Innovation, “Small Business Technology Transfer (STTR) Program—Policy Guidance,” updated February 24, 2014.

⁵ Because there is a time lag in commercialization for new technologies, the survey did not include more recent awards than 2010. See Box A-1 for a discussion of this commercialization lag.

interpreting the findings and recommendations set out below, the reader needs to keep in mind the size of the survey population and response rates, and the overall potential sources of bias.⁶

FINDINGS

The SBIR program at the DoE is having a positive overall impact. It is meeting three of the four legislative objectives of the program with regard to stimulating technological innovation, using small businesses to meet federal research and development (R&D) needs, and increasing private-sector commercialization of innovations derived from federal R&D. However, the committee finds that more needs to be done to “foster and encourage participation by socially and economically disadvantaged small businesses (SDBs), and by woman-owned small businesses (WOSBs), in technological innovation.” The STTR program at DoE is also meeting the program’s statutory objectives, defined above, in that it is encouraging and supporting linkages between small business corporations (SBCs) and research institutions (RIs).

The findings are organized according to the legislative goals for SBIR/STTR plus findings on the management of the program. The order in which the findings are presented reflects the committee’s relative emphasis. The first set of findings concerns the management of the programs at DoE. The second focuses on the commercialization of SBIR- and STTR-funded projects. This is followed by findings concerning the participation of women and minorities in the program. The fourth and fifth sets of findings address how well the DoE SBIR/STTR programs are stimulating technological innovation and fostering innovative companies. The final set of findings concerns STTR. The summary below provides a guide to the more detailed description to follow.

Summary of Findings

1. Program Management

1. DoE has substantially improved its SBIR/STTR programs since 2008 (the publication year of the previous National Academies report on the DoE SBIR program).
2. DoE has not addressed some other important recommendations from the National Academies’ 2008 report.
3. The DoE application review system can be improved.
4. The DoE SBIR/STTR programs provide companies some flexibility, but can offer more.
5. DoE is seeking ways to improve its data collection and tracking of SRIR/STTR project outcomes.

___________________

⁶ For an overview of the potential sources of survey bias, see Box A-1 in Appendix A.

2. Commercialization

1. Nearly half of the respondents to the National Academies’ 2014 Survey reported some sales, and a further 23 percent reported anticipating future sales.
2. Commercialization of DoE SBIR/STTR projects takes place primarily in the domestic private sector.
3. Subsequent investment in DoE SBIR/STTR projects is an indicator that they are seen as having the potential for commercial value even if they have not yet reached the market.
4. Direct job growth from DoE SBIR/STTR awards is in general limited, though some awardees reported large employment gains.
5. SBIR/STTR funding makes a substantial difference in determining project initiation, scope, and timing.
6. Venture capital funding plays only a modest role for DoE SBIR/STTR firms.

3. Fostering the Participation of Women and Other Underserved Groups in the SBIR/STTR Programs

1. Current data show that the objective of fostering the participation of women and underserved minorities has not been met by the DoE SBIR/STTR programs.
2. DoE efforts to “foster and encourage” the participation of woman-owned and minority-owned small businesses are not adequate.
3. DoE is making efforts to understand the patterns of woman and minority participation in the SBIR program, but more is needed.

4. Stimulating Technological Innovation and Meeting Agency Mission Needs

1. The DoE SBIR/STTR programs support the development and adoption of technological innovations that advance the agency’s mission.
2. The DoE SBIR/STTR programs continue to connect companies to universities and research institutions.
3. DoE SBIR/STTR projects generate substantial knowledge-based outputs such as patents and peer-reviewed publications.
4. The DoE SBIR/STTR programs fund some projects that have high scientific or social value, but are unlikely to generate significant market outcomes in the short term.
5. SBIR/STTR funds the development of research tools, multiplying the impact of the award.

5. Fostering Innovative Companies

1. The DoE SBIR/STTR programs encourage new firm start-up.
2. DoE SBIR/STTR funding helps small innovative companies in a variety of ways.
3. Most DoE SBIR/STTR awardees surveyed report positive impacts on their company.
4. Company dependence on the DoE SBIR/STTR programs is limited.

6. STTR

1. STTR is meeting the program objectives defined in the Small Business Administration’s Policy Guidance for STTR.
2. At DoE, the STTR program is administered as an adjunct to the much larger SBIR program.
3. Outcomes from STTR are broadly similar to those from SBIR, but there were some differences that should be noted.
4. In some cases, companies utilize STTR and SBIR differently.
5. The DoE SBIR and STTR programs are not sufficiently integrated with the DoE National Laboratories.

I. Program Management

1. DoE has substantially improved its SBIR/STTR programs since 2008 (the publication year of the previous National Academies report on the DoE SBIR program).

1. A number of recommendations from the 2008 report have been adopted:
   • DoE publishes contact information for each topic and subtopic in the solicitation.
   • DoE collects outcomes data on a regular basis and a new staff member is tasked with undertaking more systematic analysis of the program for management purposes.
   • DoE provides for a pre-release period during which technical staff can be contacted to discuss possible approaches to topics.
2. DoE has adopted a number of other initiatives and pilot programs,⁷ which collectively have improved the program. Among the more important are:

___________________

⁷ See Chapter 3 (DoE Initiatives).

3. • Integration of SBIR/STTR applications, so that applicants can, when appropriate, apply to both programs with a single application
   • Introduction of a Fast Track option to combine Phase I and Phase II into a single application
   • Introduction of sequential Phase II awards, with the Phase IIA and Phase IIB programs
   • Provision up to $15,000 for patenting expenses in approved budgets
   • Introduction of the letter of intent procedure, which provides a company with guidance about its proposed approach before it develops a full-scale application
   • Inclusion of a line item for publication costs in project budgets
4. DoE is increasing the number of Fast Track awards (combined Phase I and Phase II).
   • The number of Fast Track awards has grown steadily and in FY 2014 reached 70 awards.
   • As percentage of all regular Phase II + Fast Track awards, Fast Track awards increased from 8 percent in FY 2005 to 24 percent in FY 2014.
5. DoE has increased the emphasis on commercialization, which affects topic managers and companies.⁸
   • Commercialization plans are required for Phase I proposals, and a very detailed plan is required for Phase II.
   • Topics are now scrutinized for commercial possibility before they approved by the DoE SBIR/STTR Program Office, although it appears that more could be done in this area.
   • One case study company explained that it was excluded from the program because its commercialization record was insufficient.
6. DoE has fully embraced the benefits of electronic and electronic provision of application information and support.⁹
   • In 2008 DoE was one of the last remaining agencies to require paper applications. As with the other study agencies in this series, all applications to DoE are now electronic only.
   • DoE provides training and support through webinars, both live and archived, which reached more than 3,000 viewers during the past year.

___________________

⁸ See Chapter 2 (Program Management).

⁹ See section on “Awards Management” in Chapter 2 (Program Management).

6. • Dawnbreaker, the agency’s commercialization support contractor, provides more detailed support materials electronically.
7. The Phase 0 program piloted in FY 2014 is a promising effort to provide more support to underrepresented groups.¹⁰
8. DoE has appropriate mechanisms in place to manage issues related to the provision of multiple awards to the same company, and in general new entrants receive a substantial share of DoE SBIR/STTR funding.¹¹
   • New participants: New companies accounted for greater than 30 percent of companies applying for Phase I funding in almost all the years of the study period. New companies accounted for between 20 to 25 percent of awards and 25 to 30 percent of FY 2005-2013 applications during the study period. The access provided for new entrants to the program is appropriate.
   • Multiple award winners: The top 20 award winners at DoE accounted for 7.7 percent of Phase I SBIR/STTR awards and 8.1 percent of Phase I funding. They accounted for 9.6 percent of awards and 8.1 percent of funding for SBIR/STTR Phase II awards. This level of concentration is lower than that for other agencies. For example, at the Department of Defense (DoD) the top 20 winners accounted for 14.4 percent of awards and 14.3 percent of Phase I SBIR funding.
9. DoE has successfully shortened the gap between Phase I and Phase II funding, primarily by rearranging the schedule of solicitations and providing for dedicated contracts staff. As a result, DoE now meets the SBA guidelines for awarding and contracting Phase I and Phase II SBIR/STTR awards.¹²

2. DoE has not addressed some other important recommendations from the National Academies’ 2008 report.

1. Funding between DoE programs and divisions has not been reallocated, which leads to significant imbalances in the program.¹³
   • Currently, funding for SBIR/STTR is allocated more or less in proportion to the extramural funding by division and program, on which the SBIR/STTR funding is based.

___________________

¹⁰ See section on “Phase 0” in Chapter 3 (DoE Initiatives).

¹¹ See section on “New Entrants into the SBIR/STTR Programs and Multiple-Award Winners” in Chapter 4 (SBIR and STTR Awards and DoE).

¹² See section on “Improved Process Timelines” in Chapter 3 (DoE Initiatives).

¹³ See Chapter 2 (Program Management).

2. • This leads to equivalence between programs that operate in areas of considerable commercial opportunity and programs that focus on DoE’s science missions.
   • Because of an imbalance between the share of possible applicants and the share of available funding, EERE has eliminated “open” topics and narrowed the scope of its published technical topics, to limit the number of applicants. This is not best practice for the program, because it permanently excludes potentially important technologies.
3. Activities related to Phase III. The National Academies’ 2008 report recommended that DoE consider implementing a match-making program of some kind, perhaps modeled on the Navy Opportunity Forum.
   • The SBIR/STTR Program Office continues to see its responsibilities essentially ending at the conclusion of Phase II (or Phase IIB).¹⁴
   • A number of pilot or fully realized projects at other agencies could be the basis for more effective activity in this area.
4. The procedure for developing topics could be improved.¹⁵
   • The problem of noncommercial topics: DoE topics and subtopics are primarily the responsibility of technical staff within the science divisions and applied programs, each of which has its own procedures for validating topics.
   • There is a contradiction between the selection of topics by the science divisions in particular and the emphasis placed on commercialization. The science divisions primarily focus on SBIR/STTR to address their own technical needs; there is little evidence that they have any interest or expertise in identifying commercially important topics even within the relatively less commercial domains within which they operate.¹⁶
   • Unfunded topics: DoE’s current practice is to prioritize awards after all applications have been received.¹⁷ This places a substantial burden on companies responding to unfunded topics.
   • All projects within a division essentially compete with each other (or even across divisions—see above). As a result, some topics are not funded, even if applications for those topics are marked as fundable.
   • Some companies also complain that the same topics are regularly unfunded, which suggests that these topics are low priority and should

___________________

¹⁴ See section on “Beyond Phase II” in Chapter 2 (Program Management) and section on “Support for Improved Commercialization Outcomes” in Chapter 3 (DoE Initiatives).

¹⁵ See Chapter 2 (Program Management).

¹⁶ See Chapter 2 (Program Management).

¹⁷ See Chapter 2 (Program Management).

3. The DoE application review system can be improved.

1. Case studies, survey responses, and discussions with agency managers all indicated that, although the DoE application review system is highly regarded and has many positive characteristics, it is not serving the SBIR/STTR community as well as it could.
2. DoE reviewers act independently. They do not meet and no consensus view of a proposal is developed. Often only two reviewers are used, and about 40 percent of all reviewers are staff from the National Laboratories. This can lead to inaccurate reviews.
   • The lack of a consensus-making process means there are no checks or balances imposed on individual reviewers by other reviewers, leaving this to be provided in some cases by the topic manager.¹⁹
   • The impact of missing internal cross checks is multiplied because there is no formal procedure for resubmission whereby applicants can correct misunderstandings or provide additional data as needed. This contrasts for example with the NIH resubmission process or with the iterative approach utilized by most peer-reviewed scientific journals.²⁰
   • The impact is also multiplied by the tight competition for awards, which means that a bad score on one criterion from one reviewer is likely enough to doom a proposal.
   • The scoring system is a blunt instrument. Each application is scored on each of the four criteria as “not acceptable,” “acceptable,” or “outstanding.” This leaves no room for a more nuanced appraisal.
3. Because DoE reviews are not sufficiently transparent, the end results are sometimes viewed as unfair by applicants.²¹
   • Considerably more applications are graded as “outstanding” and therefore fundable than there are sufficient funds. As a result, 16 percent of “fundable” SBIR applications were not funded in FY 2015.
   • The process of selecting which “fundable” applications to actually fund is opaque to the applicant, because divisions apply their own priority criteria, which are not published and, in fact, may not exist in written form.

___________________

¹⁸ See Appendix E (Case Studies).

¹⁹ See Chapter 2 (Program Management).

²⁰ See Chapter 2 (Program Management).

²¹ See Chapter 2 (Program Management).

4. The DoE SBIR/STTR programs provide companies some flexibility, but can offer more.²²

1. DoE provides small amounts of supplementary funding (Phase IIA) when the completion of Phase II research plans can be accomplished with a minor increase in support.
2. DoE will normally extend the timeline for an award.
3. DoE could however offer greater flexibility to make the program more useful to small innovative companies.
   • Multiple Solicitations: Currently, DoE releases only one annual solicitation for each division-program, which means that there is one annual window through which a promising technology might be funded through SBIR/STTR.
   • Support between Phase I and II: DoE does not at present provide support between Phase I and Phase II. Even if funding is not available, other tools available could provide additional support. For example, NIH reimburses companies for work completed during this gap period if they eventually win a Phase II award.
   • Preference for lower cost proposals: Although only maximum funding amounts are set, the selection process does not give preference to companies that seek to perform research at lower cost. As a result, almost all DoE awards are made at the maximum allowable amount.
   • Gap funding program: DoE does not offer a gap funding program, such as those offered by some components at DoD. However, DoE is funding more Fast Track awards, which can solve the gap issue for those projects.
   • “Open” topics at EERE: Currently, there are no open topics in this area in an effort to limit the number of applications. Adding open topics would substantially increase the programs flexibility in accepting applications covering a wider set of technologies.

5. DoE is seeking ways to improve its data collection and tracking of SBIR/STTR project outcomes.

1. DoE recognizes that longer term tracking of outcomes is essential for effective program management: without outcomes data and analysis it is impossible to determine what is and is not working.
2. DoE has collected outcomes data from companies for some years through a survey of award recipients. The survey is deployed periodically (although not annually) and is web based. DoE did not share these survey results with the National Academies on privacy grounds.

___________________

²² See Chapter 2 (Program Management).

3. DoE has hired a data analyst to track outcomes, with primary responsibilities in this area. In addition, DoE is also improving access to its own Portfolio Analysis and Management System for analytic purposes.
4. However, DoE does not have a plan to determine which areas should be prioritized for analysis and how this analysis should be integrated into program management.

II. Commercialization

The focus at DoE has primarily been on the commercialization of SBIR- and STTR-funded projects and on the development of technologies that help to meet the agency’s mission (discussed separately below). The committee recognizes that issues related to commercialization are complex.²³ For DoE, commercialization objectives are primarily met when projects are commercially successful in private-sector markets. The key findings are as follows:

1. Nearly half of the respondents to the National Academies’ 2014 Survey reported some sales, and a further 23 percent reported anticipating future sales.

1. Forty-nine percent of SBIR and STTR respondents reported some sales at the time of the survey. ²⁴ By comparison, 39 percent of the National Academies’ 2005 Survey respondents reported some sales.²⁵
2. An additional 23 percent of respondents reported that they anticipate future sales,²⁶ which is greater than the 16 percent figure found in the 2005 Survey.²⁷
3. Of those respondents reporting some sales, 25 percent had sales less than $100,000. Six percent had sales over $10 million, and an additional 26 percent had sales over $1 million.²⁸ The large number of companies with small-scale revenues suggests that although many companies reach the market, few can be described as successful in commercial terms. This finding reflects a deeper understanding of the limitations of the available data on successful commercialization.

___________________

²³ See the discussion on “Defining ‘Commercialization’” in Appendix F.

²⁴ See Figure 5-1.

²⁵National Research Council, An Assessment of the SBIR Program at the Department of Energy, Washington, DC: The National Academies Press, 2008, p. 143.

²⁶ See Figure 5-1.

²⁷National Research Council, An Assessment of the SBIR Program at the Department of Energy, Washington, DC: The National Academies Press, 2008, p. 143.

²⁸ See Figure 5-2.

2. Commercialization of DoE SBIR/STTR projects takes place primarily in the domestic private sector.²⁹ Among surveyed projects reporting sales—

1. An average of 39 percent of project sales were to domestic private-sector customers.³⁰
2. An average of 24 percent of project sales were to export customers.³¹
3. An average of 29 percent of sales were to the public sector (primarily state and local governments or other federal agencies).³²
4. Only an average of 6 percent of reported sales were to DoE.³³

3. Subsequent investment in DoE SBIR/STTR projects is an indicator that they are seen as having the potential for commercial value even if they have not yet reached the market.³⁴ The 2014 Survey shows that:

1. Seventy-eight percent of 2014 Survey respondents reported receiving additional investment funding in the technology related to the surveyed project.³⁵
2. Overall, the median amount of additional investment was $300,000, and the mean was $814,000. Three out of 245 projects reported additional investments of at least $5 million.³⁶
3. The most likely sources of additional investment (other than their own company and personal funds) were non-SBIR/STTR federal funds and the U.S. private sector (reported by 40 percent and 39 percent of respondents respectively). The latter included funding from strategic investors (reported by 20 percent of respondents), angel investors (reported by 5 percent of respondents), and venture capital sources (reported by 2 percent of respondents).³⁷

4. Direct job growth from DoE SBIR/STTR awards is in general limited, though some awardees reported large employment gains. ³⁸

1. The median size of firms responding to the 2014 Survey remained flat at 10 employees between the time of award and the time of the survey.³⁹ All other things being equal, larger employment gains are more

___________________

²⁹ See Table F-3.

³⁰ See Table F-3.

³¹ See Table F-3.

³² See Figure 5-3.

³³ See Figure 5-3.

³⁴ See Table 5-2.

³⁵ See Table 5-2.

³⁶ See Table 5-2.

³⁷ See Table 5-3.

³⁸ See Tables F-4 and F-5.

³⁹ See Tables F-4 and F-5.

2. typically associated with the long-term commercialization phase of the resulting innovation, rather than with the research phase.

3. Few firms reported large employment gains: mean employment grew only from 23 to 29.⁴⁰

5. SBIR/STTR funding makes a substantial difference in determining project initiation, scope, and timing. The 2014 Survey data show that:

1. Seventy-one percent of respondents reported that the project probably or definitely would not have proceeded without SBIR/STTR funding.⁴¹
2. Seventy-four percent of projects that would likely have proceeded anyway would have been narrower in scope.⁴²
3. About one-third of projects that would likely have proceeded anyway would have been delayed by at least 1 year.⁴³

6. Venture capital funding plays only a modest role for DoE SBIR/STTR firms.

1. Although 78 percent of companies raised additional funds for the technology related to the surveyed project,⁴⁴ only 2 percent received funds from venture capitalists (VCs).⁴⁵
2. Energy technologies featured in DoE SBIR/STTR projects typically do not meet the narrow criteria sought by VC firms, including a short timeline to market exit and limited size of funding required.

III. Fostering the Participation of Women and Other Underserved Groups in the SBIR/STTR Programs

1. Current data show that the objective of fostering the participation of women and underserved minorities has not been met by the DoE SBIR/STTR programs.

1. DoE awards data reveal that the participation of woman-owned firms is low and not growing:
   • Woman-owned firms accounted for less than 9 percent of Phase I SBIR and STTR awards in FY 2005-2015.⁴⁶ The average success

___________________

⁴⁰ See Tables F-4 and F-5.

⁴¹ See Table 5-1.

⁴² See section on “Project Go-ahead Absent SBIR/STTR Funding” in Chapter 5 (Quantitative Outcomes).

⁴³ 2014 Survey, Question 26. N=27.

⁴⁴ See Table 5-2.

⁴⁵ See Table 5-3.

⁴⁶ See Figure 6-3.

2. • rates for Phase I applications by firms owned by woman and white males were 15.7 percent and 18.9 percent, respectively, during this period.⁴⁷

   • Applications from woman-owned firms accounted for an average of 10 percent of all Phase I applications during FY 2005-2015.⁴⁸
   • Among respondents to the 2014 Survey, women accounted for 5 percent of SBIR PIs and 10 percent of STTR PIs.⁴⁹
3. The participation of minority-owned small businesses is low and not growing.
   • Minority-owned firms accounted for less than 7 percent of Phase I SBIR and STTR awards during FY 2005-2015.⁵⁰
   • The average success rates for Phase I applications by firms owned by minorities and white males were 13.2 percent and 18.9 percent during this period.⁵¹
   • Applications from minority-owned firms accounted for 5.8 percent of all Phase II applications during FY 2005-2015.⁵²
   • Among respondents to the 2014 Survey, the vast majority of “minority” firms were in fact owned by Asians. Firms owned by Blacks, Hispanics, and American Indians accounted for 2 percent of all responses (including zero Black-owned and American-Indian owned firms).⁵³
   • Among respondents to the 2014 Survey, minorities accounted for 11 percent of SBIR PIs and 10 percent of STTR PIs. However, no PIs were Black or American Indian, and only 2 percent overall were Hispanic.⁵⁴

2. DoE efforts to “foster and encourage” the participation of woman-owned and minority-owned small businesses are not adequate.⁵⁵

1. DoE outreach efforts have focused more heavily on attracting participation from low-award states than from women-owned and minority-owned small businesses.
   • The SBA-sponsored Road Show is a primary outreach activity but focuses on low-award states.

___________________

⁴⁷ See Figure 6-5.

⁴⁸ See Figure 6-1.

⁴⁹ See Table 6-5.

⁵⁰ See Figure 6-4.

⁵¹ See Figure 6-6.

⁵² See Figure 6-8.

⁵³ See Table 6-4.

⁵⁴ See Table 6-5.

⁵⁵ See Chapter 6 (Participation of Women and Minorities).

2. • The DoE Annual Report to SBA for FY 2014 mentions a considerable catalog of outreach activities—but mentions underrepresented groups only as a part of one activity.
   • Most DoE outreach is conducted in conjunction with other partners. This means that DoE has limited capacity to modify these events to address its own needs. DoE is working to improve reporting on outreach activities with these partners, especially in relation to women and minorities.
3. DoE is developing outreach activities focused on women and minorities. In particular, DoE is planning to work more closely with organizations serving female and minority professionals.

3. DoE is making efforts to understand the patterns of woman and minority participation in the SBIR program, but more is needed. ⁵⁶

1. DoE maintains no separate data on small businesses owed by Black Americans, Hispanic Americans, or Native Americans beyond those collected by Dawnbreaker for Phase 0.
2. DoE has not reviewed application and award patterns for women and minorities in detail. These patterns could show differences between woman and minority applications and other applications on a variety of metrics.
3. DoE has sought to contextualize observed trends and patterns of participation by woman- and minority-owned firms against larger patterns of participation in the energy sector. This can help DoE determine whether low participation rates are a function of the energy sector, the SBIR/STTR programs, the financial and business communities, or a combination of these factors.
4. DoE prepared a white paper on the population of woman- and minority-owned businesses in research-heavy sectors—as categorized by the North American Industry Classification System (NAICS)—in 2013.
5. The DoE SBIR/STTR Program Office is undertaking an innovative analysis that utilizes company records in the System for Awards Management (SAMS) database in order to better identify specific NAICS codes relevant to the energy sector. Once complete, this analysis will allow DoE to cross-reference Census business establishment data for these sectors in order to determine the percentages of woman- and minority-owned firms within these

___________________

⁵⁶ See Chapter 6 (Participation of Women and Minorities).

6. NAICs codes. This may turn out to be a valuable project and to provide a methodology that may be applicable to other agencies.

7. However, this work aims only to develop appropriate benchmarks. It does not improve outreach; and focuses only on existing firms, which means that it does not address the need to find applicants who have not yet formed companies, a population that is important in the context of SBIR/STTR.

IV. Stimulating Technological Innovation and Meeting Agency Mission Needs

DoE’s agency mission is to enhance “America’s security and prosperity by addressing its energy, environmental, and nuclear challenges through transformative science and technology solutions.”⁵⁷ The twin objectives of using small business to meet federal agency needs and to stimulate technological innovation are closely intertwined and therefore discussed together in this section.

1. The DoE SBIR/STTR programs support the development and adoption of technological innovations that advance the agency’s mission.

1. SBIR/STTR topics are initially generated by DoE technical staff (Technical Topic Managers, or TTMs) based on their perception of the technical needs of the programs in which they work.⁵⁸
2. In some cases, topics are designed to generate technologies that are for DoE’s direct use, usually in the National Laboratories.
3. “Open” topics, which permit submission of applications that are not limited to problems and technologies described in the DoE solicitations is now standard practice for all DoE science divisions and applied programs except the Office of Energy Efficiency and Renewable Energy (EERE). This approach helps to ensure that potentially important innovations are not excluded by the topic structure, and open topics account for about 7 percent of awards.
4. DoE scoring selects for innovation.⁵⁹
   • The first criterion used in selection scoring for individual projects is “the significance of the technical and-or economic benefits of the proposed work, if successful.”
   • Qualitative research confirms that, in practice, SBIR/STTR review focuses heavily on, and provides detailed critiques of, the technical quality of proposals.

___________________

⁵⁷ See http://energy.gov-mission. Accessed February 25, 2016.

⁵⁸ See section on “Solicitation Topics” in Chapter 2 (Program Management).

⁵⁹ See section on “The Review Process” in Chapter 2 (Program Management).

2. The DoE SBIR/STTR programs connect companies to universities and research institutions.

1. Faculty and student participation: Among SBIR awardees responding to the 2014 Survey,
   • 43 percent reported a link to a research institution related to the surveyed project;
   • 26 percent reported that faculty worked on the project (not as a PI);
   • 21 percent employed graduate students for the project; and
   • 29 percent used universities and research institutions as subcontractors for the surveyed project. ⁶⁰ These percentages are broadly similar to those reported for the 2005 Survey.⁶¹
2. Research institutions are important project partners: 79 different research institutions were identified by 2014 Survey respondents as partners in 148 total projects; 15 were mentioned by three or more respondents.⁶²
3. More than two-thirds of companies responding to the 2014 Survey reported at least one academic founder, ⁶³ and just under one-third reported that the most recent prior employer of a founder was at a research institutions.⁶⁴

3. DoE SBIR/STTR projects generate substantial knowledge-based outputs, such as patents and peer-reviewed publications.

1. Patents: Patenting remains an important component of knowledge diffusion (and protection).
   • Sixty-eight percent of companies responding to the 2014 Survey reported receipt of at least one patent related to any SBIR/STTR-funded technology.⁶⁵

___________________

⁶⁰ See Table F-13.

⁶¹ National Research Council, An Assessment of the SBIR Program at the Department of Energy, 153.

⁶² See Appendix D (List of Research Institutions Involved in Surveyed DoE SBIR/STTR Awards).

⁶³ See Table F-15.

⁶⁴ See Table F-16.

⁶⁵ See Table F-10.

2. • Thirty-nine percent of SBIR/STTR respondents reported receipt of at least one patent related to the surveyed technology.⁶⁶
3. Peer-reviewed publications: The 2014 Survey shows that DoE SBIR/STTR firms continue to pursue and achieve scientific publication.
   • Seventy-one percent of SBIR respondents and 88 percent of STTR respondents indicated that an author at the surveyed company had published at least one related scientific paper.⁶⁷
   • Overall, 39 percent of respondents reported publishing three or more related papers.⁶⁸
   • Many of the case study companies reported a great deal of pride in the number of peer-reviewed publications developed by their scientists and engineers, both inside and outside of the SBIR/STTR programs.⁶⁹

4. The DoE SBIR/STTR programs fund some projects that have high scientific or social value, but are unlikely to generate significant market outcomes in the short-term.

1. Small markets: The science divisions of the Department of Energy, in particular, sponsor projects that are not likely to generate large-scale commercial returns. For example, only a few high-energy physics facilities exist in the world, so technologies to support them offer little commercial return despite substantial social and/or scientific impact.
2. Long-cycle research: DoE SBIR/STTR helps support the development of innovations that will take many years to reach the market. Although these projects may hold great potential for positive impact, the time to get to market can be a major barrier for commercial investors. This has been as important characteristic of innovation in energy markets.

5. SBIR/STTR funds the development of research tools, multiplying the impact of the award.

1. Case studies show that the impact of awards can be multiplied if SBIR/STTR technologies are used to develop innovative tools and services for researchers.⁷⁰
2. XIA, for example, produces instruments that are critical to the effective functioning of large accelerators such as the Stanford Linear Accelerator Center (SLAC). Without the relatively modest production of these

___________________

⁶⁶ See Table F-11.

⁶⁷ See Table F-12.

⁶⁸ See Table F-12.

⁶⁹ See Appendix E (Case Studies).

⁷⁰ See Appendix E (Case Studies).

V. Fostering Innovative Companies

1. The DoE SBIR/STTR programs encourage new firm start-up.

1. Forty-five percent of companies responding to the 2014 Survey indicated that the company was founded entirely or in part because of the SBIR/STTR programs.⁷²
2. For some case study companies, SBIR/STTR funding permitted the shift from an exploratory to a professional operation. For some STTR companies in particular, funding permitted university faculty to retain their positions while building the company.⁷³

2. DoE SBIR/STTR funding helps small innovative companies in a variety of ways.

1. Early stage: Several survey respondents and case study companies explained that DoE SBIR/STTR funding was provided at a stage when the project was simply too risky for commercial sources of funding. Once the project proceeded further, the risk diminished and additional funding could be acquired.⁷⁴
2. Support for core technology development: DoE SBIR/STTR funding supports technology development, which can be supported through commercial funding further downstream. SBIR/STTR is particularly important for funding proof of concept for new technologies, as described in several case studies as well as in survey responses.⁷⁵
3. Validation and certification effects: DoE SBIR/STTR funding has itself provided important validation for companies seeking further investments, according to discussions with representatives from case study companies and survey responses. The strength of the selection process and growing understanding of SBIR/STTR among both equity and strategic investors may be strengthening this effect.⁷⁶
4. Exploit technology platforms: In some cases, companies use SBIR/STTR funding to build off existing platform technologies specifically to enter new markets. This platform-driven approach is used for example by some companies highlighted in the case studies.⁷⁷

___________________

⁷¹ See XIA, LLC case study in Appendix E.

⁷² See Table F-17.

⁷³ See Appendix E (Case Studies).

⁷⁴ See Chapter 7 (Insights from Case Studies and Survey Responses).

⁷⁵ See Chapter 7 (Insights from Case Studies and Survey Responses) and Appendix E (Case Studies).

⁷⁶ See Chapter 7 (Insights from Case Studies and Survey Responses).

⁷⁷ See Appendix E (Case Studies).

3. Most DoE SBIR/STTR awardees surveyed report positive impacts on their company.

1. Sixty-one percent of respondents to the 2014 Survey indicated that the DoE SBIR/STTR programs “had a highly positive or transformative effect” on their company. Another 35 percent said that it “had a positive effect.”⁷⁸
2. Box 8-1 lists the many types of impacts, summarized from the 217 detailed comments received in response to the 2014 Survey.

4. Company dependence on the DoE SBIR/STTR programs is limited.

1. DoE limits the number of applications from a single company to 10 per solicitation, which is similar to the approach taken by the National Aeronautics and Space Administration.⁷⁹
2. Awards are spread widely across the applicant pool.⁸⁰
3. The company’s commercialization track record is of growing importance. For example, Dr. Johnson (Muons Inc.) explained that his company’s poor commercialization record made it ineligible for DoE SBIR/STTR awards; after becoming more commercial, it started to receive awards again. Clearly, DoE’s commercialization requirements have some teeth.⁸¹
4. Most DoE firms are not dependent on SBIR/STTR awards. Only 25 percent of companies responding to the 2014 Survey reported that SBIR/STTR accounted for greater than one-half of revenues for the most recent fiscal year at the time of the survey, while greater than one-quarter had zero SBIR/STTR-related revenues for the same period.⁸² However a considerable number of surveyed firms reported in textual responses that SBIR/STTR was the most important funding source prior to reaching the market.
5. DoE SBIR/STTR projects at many companies do not proceed directly from Phase I to Phase II to commercialization and, as a result, additional funding—which often includes further SBIR/STTR funding—may be needed to reach the market.

___________________

⁷⁸ See Table F-22.

⁷⁹ See Chapter 2 (Program Management).

⁸⁰ See Chapter 2 (Program Management).

⁸¹ See case study of Muons, Inc. in Appendix E.

⁸² See Table F-19.

• Sixty-three percent of 2014 Survey respondents reported at least one additional SBIR/STTR Phase II award related to the surveyed project.⁸³
• Twenty-eight of respondents reported at least two additional Phase II awards related to the surveyed project.⁸⁴

VI. STTR

1. STTR is meeting the program objectives defined in the Small Business Administration’s Policy Guidance for STTR.

1. STTR is stimulating technological innovation, as evidenced by the substantial knowledge effects identified in Chapter 5 and the relevant case studies referenced in Chapter 7.
2. STTR fosters cooperative R&D between universities and other research organizations and industry.
   • Thirteen percent of STTR awardees responding to the 2014 Survey reported that the PI was a faculty member at the partnering research institution.⁸⁵
   • Some companies profiled in this report as case studied indicated that STTR helped to bridge the gap between research laboratories and commercial activities.⁸⁶
3. STTR at DoE is meeting the objective of supporting the commercialization of federally funded technologies.
   • Thirty-three percent of STTR awardees responding to the 2014 Survey reported sales from the surveyed project, and a further 43 percent anticipated sales in the future.⁸⁷
   • More DoE STTR awardees than SBIR awardees responding to the 2014 Survey reported additional investment in the technology aside from program funds (86 percent and 78 percent, respectively).⁸⁸

2. At DoE, the STTR program is administered as an adjunct to the much larger SBIR program.⁸⁹

___________________

⁸³ See Table F-21.

⁸⁴ See Table F-21.

⁸⁵ See Table F-13.

⁸⁶ See, for example, case study of Calabazas Creek Research in Appendix E.

⁸⁷ See Table F-1.

⁸⁸ See Table 5-2.

⁸⁹ See Chapter 2 (Program Management).

1. Discussions with DoE staff confirm that the agency jointly operates the SBIR and STTR programs, with minor differences in participation rules.
   • Solicitations for STTR and SBIR are announced jointly.
   • Companies can apply jointly for both programs.
   • The agency does not have different strategic objectives for the two programs.

3. Outcomes from STTR are broadly similar to those from SBIR, but there were some differences that should be noted.

1. Based on 2014 Survey data, STTR shows substantially greater levels of connection to research institutions than does SBIR, across most categories.⁹⁰
2. Regarding commercialization outcomes, STTR awardees responding to the 2014 Survey were less likely than SBIR awardees to claim that products had already reached the market (33 versus 52 percent). At the same time, they were more likely to claim that the product would reach the market in the future (43 versus 20 percent).⁹¹
3. Regarding knowledge effects, SBIR awardees responding to the 2014 Survey reported fewer patents related to any SBIR or STTR award received by the company than did responding STTR awardees (66 percent compared with 86 percent).⁹²

4. In some cases, companies utilize STTR and SBIR differently.

1. STTR permits PIs to spend less than 51 percent of their time on the funded project. SBIR does not. As a result, PIs who wish to retain a half-time position or more at a research institution find STTR to be a preferable option.
2. STTR also permits subcontracting a larger share of the award to the research institution, which is useful when a company requires specialized equipment or skill sets.

5. The DoE SBIR and STTR programs are not sufficiently integrated with the DoE National Laboratories.

1. Collaborating with the National Laboratories can be challenging for small innovative businesses.

___________________

⁹⁰ See Table F-13.

⁹¹ See Table F-1.

⁹² See Table F-10.

2. • Analysis of STTR in particular suggests that National Laboratories generally do not make good formal partners for small business concerns; their administrators do not prioritize SBIR/STTR because the funding amounts are small; and small businesses have limited leverage if the Laboratories fail to meet their obligations.
   • The Laboratories are viewed as bureaucratic and unsympathetic to the needs of small businesses.
   • The Laboratories also present difficulties from an IP perspective, because their culture is dominated by open academic exchange, which may not accommodate SBC desire to protect commercially important information.⁹³
   • Even though National Laboratories staff play a significant role in developing topics for SBIR/STTR, the Laboratories have no formal process for utilizing SBIR/STTR technologies, and some companies say that selling into the Laboratories is difficult.⁹⁴
3. There is potential for more fruitful collaboration between SBIR and STTR awardees and the National Laboratories.
   • A substantial share of STTR awards are made to collaborations that include the National Laboratories.
   • Case study evidence suggests that there can be fruitful collaborations between small business concerns and the National Laboratories, especially when company founders have deep understanding of and connection to the Laboratories.⁹⁵
   • Several companies reported successful engagements with the Laboratories when the latter acted as subcontractor for SBIR awards.⁹⁶
   • Collaborations can be driven by the SBC, which requires expertise or equipment from the Laboratory, or by the Laboratory, which seeks to commercialize a new technology. Although there are no hard data, evidence from case studies suggests that the former is the dominant kind of arrangement.⁹⁷
4. The DoE SBIR and STTR programs have not made sufficient efforts to enhance collaborations between the National Laboratories and small innovative firms.
   • There is no program in place to connect SBIR and STTR companies to other opportunities at the National Laboratories, even though National

___________________

⁹³ See, for example, case study of XIA, LLC in Appendix E.

⁹⁴ See, for example, case study of Vista Clara in Appendix E.

⁹⁵ See, for example, case study of Muons, Inc. in Appendix E.

⁹⁶ See, for example, case study of Calabazas Creek Research in Appendix E.

⁹⁷ See Chapter 2 (Program Management).

RECOMMENDATIONS

Although the DoE SBIR/STTR programs generate substantially positive outcomes, the committee has identified a series of recommendations to improve their processes and outcomes. The order of these recommendations reflects the relative emphasis of the committee. The first set of recommendations addresses improving monitoring, evaluation, and assessment. The second set addresses the challenge of drawing more woman- and minority-owned companies into the programs. The third set of recommendations focuses on ways to improve the commercialization of SBIR/STTR projects, followed by recommendations related to the National Laboratories, and then to program management more generally. A detailed description follows the summary of key points below.

Summary of Recommendations

1. Improving Monitoring, Evaluation, and Assessment

1. DoE should improve current data collection approaches and methodologies.
2. DoE should ensure that the outcomes data it now collects are systematically employed to guide program management.
3. DoE should prepare a comprehensive SBIR/STTR Annual Report that replaces current reporting requirements and provides a clear picture of program operations to the Secretary of Energy, Congress, and the public.

2. Addressing Underserved Populations

1. Quotas are not necessary.
2. DoE should accelerate its efforts to develop new benchmarks and metrics.
3. DoE should develop an outreach and education program focused on expanding participation of underserved populations.
4. DoE should review selection procedures and remove any identified biases in the selection process.

___________________

⁹⁸ See Chapter 2 (Program Management).

3. Improving Commercialization Outcomes

1. DoE should support the commercialization of SBIR and STTR supported technologies beyond the completion of Phase II.
2. DoE should review the effectiveness of its commercialization support and training initiatives.

4. Improving Linkages to National Laboratories

1. DoE should seek to develop programs linking Laboratories’ procurement actions with relevant SBIR/STTR projects.
2. DoE should seek ways to ensure that Laboratories fully understand and respect the intellectual property (IP) provisions of SBIR/STTR.
3. DoE should examine from a strategic perspective how the relationship of SBIR/STTR with the National Laboratories works today.

5. Improving Program Management

1. DoE should improve its topic development process.
2. DoE should change the balance of funding to better reflect innovation and commercialization opportunities in the private sector.
3. Although commercialization is an important program objective, DoE should not treat it as the only objective of the SBIR/STTR programs.
4. DoE should review and possibly rethink the relationship between the National Laboratories and SBIR/STTR.
5. DoE should improve its application review system and monitor the profile of applicants.
6. DoE should further address the funding gap between Phase I and II awards.

I. Improving Monitoring, Evaluation, and Assessment

The development of more monitoring and more sophisticated analysis of key variables is necessary to improve program outcomes. Although DoE recognizes the need for better data and is working to improve tracking mechanisms, more remains to be done in this area.

1. DoE should improve current data collection approaches and methodologies.⁹⁹

1. DoE should improve data collection and organization.
   • Data collection should be ongoing rather than periodic.

___________________

⁹⁹ See Finding I-E.

2. • Data collection should address the entire range of congressionally mandated outcomes, not just commercialization, and should be extended to other aspects of the program, including demographic data for applicants and awardees.
3. DoE should expand tracking of commercialization outcomes.
   • DoE should track commercialization outcomes drawing on a complement of metrics in order to provide a deeper and more nuanced basis for analysis.
   • The data collection effort now under way at SBA may provide DoE with additional capabilities.
4. DoE should collect enhanced demographic data.
   • DoE should take immediate steps to improve its collection of demographic data about PIs. Although DoE has explained that the provision of such data is voluntary under current federal mandates, DoE can encourage applicants to provide these valuable data.
   • DoE should extend its collection of the demographics of company ownership to show which of SBA’s socially and economically disadvantaged categories an applicant company belongs to. Although the provision of these data is also voluntary, in this case, some data is better than no data. DoE does collect these data through Dawnbreaker in relation to the pilot Phase 0 program.
5. DoE should also develop and adopt a more systematic and critical approach to the use of detailed case studies and success stories.
   • Case studies—written by DoE staff or third parties—can describe the roles played by SBIR/STTR awards, the challenges faced by small businesses, insights into needed improvements in process, lessons learned, and other important information not available elsewhere about program impacts. Currently, the available case studies are limited in number and scope and do not provide an effective review of program successes.
   • Success stories—provided by the companies—can provide inspiration and promote interest in the program, but should not be regarded as evidence of program effectiveness.
6. DoE should take advantage of modern information management and data visualization tools to communicate with companies about program activities and operations and to facilitate networking of program participants.

2. • DoE should explore ways to use new technology such as social media to collect more current data. SBIR/STTR companies—like “customers” in other markets—are an important source of information about program strengths and weaknesses. This knowledge is currently not systematically included in internal program evaluation by the DoE SBIR/STTR programs.
3. DoE should develop appropriate feedback tools for applicants and awardees. Although DoE program staff talk to participating companies daily, it has not collected systemic feedback. DoE should develop pathways for companies to provide feedback about program activities and operations. These should include various electronic communication tools.

2. DoE should ensure that the outcomes data it now collects are systematically employed to guide program management.¹⁰⁰

1. DoE should develop a plan for data analysis: The agency should seek to develop a more sophisticated approach to analyzing and applying the data that are already collected, to ensure that congressional objectives are being met, provide a data-driven basis for program improvements, and analyze outcomes from pilot initiatives.
2. DoE should in particular seek data that will help to identify factors that tend to encourage successful transitions between Phases, into Phase IIB, and then into full-scale commercialization.
3. These more comprehensive data can be the basis for addressing a range of key program management issues, such as:
   • What is the long-term impact of commercialization training, partnership programs, and other commercialization supports?
   • Is Phase IIB simply selecting successful companies or is it at least, in part, causing companies to be successful?
   • Are some National Laboratories better partners than others? Are there Laboratories that could provide best practices in this area?
   • How well do DoE selection processes predict eventual successful projects?
   • How well do commercialization plans presented in applications track in outcomes? Are they related, and, if not, then should the plan be changed or even eliminated (for either Phase I or Phase II)?
   • How effectively do initiatives such as direct to Phase II, Phase 0, and Phase IIB improve outcomes?

___________________

¹⁰⁰ See Finding I-E.

2. • How do applications from woman- and minority-owned firms transition through the SBIR/STTR pipelines into eventual commercialization? How does this compare to other applications?
3. DoE should recognize the impacts of effective data collection and analysis. In some cases, simply measuring something more closely can provoke needed action. For example, closely tracking the participation of women and minorities could help assure a fair process and surface problem issues early, when they can be most easily corrected.

3. DoE should prepare a comprehensive SBIR/STTR Annual Report that replaces current reporting requirements and provides a clear picture of program operations to the Secretary of Energy, Congress, and the public.¹⁰¹

1. New annual report: The imposition of new reporting burdens on the DoE SBIR/STTR programs does not come without cost; however, an annual report to Congress could improve transparency and provide a coherent point of discussion for stakeholders.
2. Although the precise details should be left to the agency, DoE should consider discussion of the following areas of program operations in the annual report:
   • Program Inputs: Budget and related resources input at the program’s front end.
   • Program Outputs: Initiatives developed, outreach conducted, competitions-solicitations held, applications-proposals received, awards and contracts made.
   • Program Results:
     • Early outcomes: Progress measures such as attraction of additional funding by awardees, formation of partnerships, early sales, patents, publications, and licensing agreements.
     • Intermediate outcomes: Resulting company growth in sales, employment, and knowledge benefits through the citation of patents and publication.
     • Long-term outcomes: Measures of the economic return on investment, improvements in national innovation capacity, gains in strength of small businesses attributed to the programs, and growth in the numbers and percentage of women and minority businesses comprising the SBIR/STTR client base.

___________________

¹⁰¹ See Finding I-E.

2. • • Qualitative review, based on improved use of case studies, as well as success and failure stories and social media.
   • Impact assessment, focused on the extent to which DoE meets congressional objectives for the program.
   • Summary conclusions, including prospective views on program activities and improvements for the coming year.
3. Congress should take steps to support this development by consolidating existing reporting demands into the proposed new framework. The new Annual Report should replace all existing reporting required from the program.

II. Addressing Underserved Populations

In light of the data presented in Chapter 6 and summarized in section III of the findings above, DoE should immediately extend past and current efforts to foster the participation of underserved populations in the SBIR/STTR programs, develop an outreach and education program focusing on these populations, and create benchmarks and metrics to relate the impact of such activities.¹⁰²

1. Quotas are not necessary. ¹⁰³ While DoE should strive to increase participation of under-represented populations in the SBIR/STTR programs, it should not develop quotas for that purpose.
2. DoE should accelerate its efforts to develop new benchmarks and metrics.¹⁰⁴

1. Improve participation metrics: DoE should complete and publish its current work on mining the System for Award Management (SAM) database to define appropriate NAICs codes, which can then be matched to Census data to estimate the population of woman- and minority-owned firms that constitute the pool of potential SBIR/STTR applicants.
   • DoE’s previous work provides a useful basis for the current effort.
   • If successful, the work now under way may have applications for other agencies.
   • DoE should ensure that its data collection and analysis apply to all congressional objectives, not only commercialization.

___________________

¹⁰² See Finding III.

¹⁰³ See Box 6-3.

¹⁰⁴ See Finding III-C.

2. Disaggregate benchmarks: Measures of the participation of socially disadvantaged groups should be disaggregated by race or ethnicity, and attention should be focused on the congressional intent to support “minority” participation. The current SBA definition of “socially and economically disadvantaged” is not sufficient to meet this objective.
3. Customize benchmarks: Points of reference should be developed separately (though perhaps drawing on a shared methodology) for women and minorities. Benchmarks should address the following metrics, all of which should include both absolute levels and trends over time:
   • Shares of applications from companies owned by women and minorities.
   • Shares of applications with woman and minority PIs.
   • Shares of Phase I awards to companies owned by women and minorities.
   • Shares of Phase I awards with female and minority PIs.
   • Shares of Phase II awards to companies owned by women and minorities.
   • Shares of Phase II awards with female and minority PIs.
4. Track related program operations: Metrics should also track related program operations including outreach efforts (see below).

3. DoE should develop an outreach and education program focused on expanding participation of underserved populations.¹⁰⁵

1. DoE’s new Phase 0 program is a promising initiative. However, its initial focus on supporting existing applicants and on under-served states is not sufficient to meet the need for enhanced outreach.
2. Develop enhanced outreach strategy: DoE should develop a coherent and systematic outreach strategy that provides for cost-effective approaches to enhance recruitment of woman- and minority-owned companies, as well as female and minority PIs, developed in conjunction with other stakeholders and experts in the field. Outreach should aim to expand SBIR/STTR awareness among potential applicants from underserved demographics.
3. Add-ons to existing outreach activities are not sufficient. There is no evidence that a panel at the national SBIR conference has attracted significant numbers of new participants from target demographics (or indeed any demographics) into the program. The SBA Bus Tour is

___________________

¹⁰⁵ See Finding III-B.

4. targeted more at reaching potential applicants in underserved states than attracting women and minorities to the program. Focused and extensive outreach activities will be needed.

5. Provide management resources: DoE should provide significant management resources, because these outreach efforts are likely to be difficult and long term, and should consider designating a senior staff member to work exclusively on outreach to women and minorities to improve reporting and deployment of the new initiatives.

4. DoE should review selection procedures and remove any identified biases in the selection process. ¹⁰⁶

1. Review selection processes: DoE should review internal award and selection data and processes to address questions arising from disparities between Phase I and Phase II success rates for woman- and minority-owned firms and firms not in those categories. The goal is to ensure that there are no biases in the selection process that adversely affect the selection of women and minorities.
2. Monitor selection processes: DoE should ensure that patterns of applications, awards, and success rates are monitored and reported out annually.
3. DoE should ensure that reviewers include appropriate numbers of women and minorities. One additional reason to increase the number of reviewers is to expand this pool.

III. Improving Commercialization Outcomes

The DoE SBIR/STTR programs are fulfilling their commercialization mission despite the substantial barriers to commercializing innovative research. However, possible improvements are worth consideration.

1. DoE should support the commercialization of SBIR and STTR supported technologies beyond the completion of Phase II.¹⁰⁷

1. DoE is well positioned to support company commercialization efforts. Without undertaking resource-intensive efforts, it is still possible to develop programs that support companies in this critical area. For example:
   • DoE could undertake its own version of the Navy Opportunity Forum to connect SBIR/STTR companies with investors and strategic

___________________

¹⁰⁶ See Finding III-B.

¹⁰⁷ See Finding I-B.

• partners, or find ways to work with existing conferences as appropriate (as, for example, NIH does with BIO conferences).

• DoE could adopt the online approaches used by the Air Force and NASA to develop attractive and searchable databases of companies and technologies, which could be publicized to potential investors and partners.

2. DoE should review the effectiveness of its commercialization support and training initiatives.¹⁰⁸

1. While DoE should be commended for providing commercialization support on a regular basis to both Phase I and Phase II SBIR/STTR awardees, it should consider whether current commercialization support is effective. By the time the Dawnbreaker contract expires in March 2017, DoE should have developed and implemented a methodology for assessing the impact of Dawnbreaker’s work.
2. More generally, the DoE SBIR/STTR Program Office should review initiatives to assess apparent successes for potential expansion and to learn from failures.
3. More broadly, DoE should consider whether its current approach to require cost sharing for downstream demonstration (non-SBIR/STTR) projects outside the Office of Science is appropriate for small innovative firms. High cost-sharing requirements are a barrier to participation for these firms.

IV. Improving Linkages to National Laboratories¹⁰⁹

The National Laboratories perform multiple functions in relation to SBIR/STTR. Their staff generate many of the topics, account for about 40 percent of reviewers, provide important access to expertise and equipment for both SBIR and STTR projects, are formal partners for many STTR projects; and are significant customers for some SBIR/STTR technologies. The following recommendations are predominantly for DoE senior management, rather than SBIR/STTR Program Office staff.

1. DoE should seek to develop programs linking Laboratories’ procurement actions with relevant SBIR/STTR projects.

1. Significant sole sourcing advantages of SBIR/STTR awards should be highlighted to Laboratory management and contracts officers.
2. DoE should review DoD procedures for ensuring that SBIR/STTR topics are sponsored by acquisition elements of the agency. Although this will

___________________

¹⁰⁸ See Finding I-B.

¹⁰⁹ Recommendations in this section are based on the analysis summarized in findings in section VI-E.

2. DoE should seek ways to ensure that Laboratories fully understand and respect the intellectual property (IP) provisions of SBIR/STTR.

1. Although formal IP clauses are always part of STTR partnerships agreements, and are often part of SBIR subcontracts, DoE should ensure that staff acting as technical points of contact for these agreements understand and agree with these provisions.
2. DoE should review its procedures in cases when IP is covered by an agreement with an SBC but is not treated appropriately.

3. DoE should examine from a strategic perspective how the relationship of SBIR/STTR with the National Laboratories works today.

1. The review should explore how these relationships could be improved.
2. Should DoE conclude that the relationships incurs more costs than benefits, it should seek to reduce the relationship along several dimensions (e.g. using more external reviewers, reducing incentives to partner with National Laboratories, reducing the use of SBIR/STTR to serve the specific scientific objectives of Laboratories’ staff at the expense of commercialization).

V. Improving Program Management

The following recommendations are designed to improve program operations in ways that should enhance the program’s ability to address some or all of its objectives.

1. DoE should improve its topic development process.¹¹⁰ The current topic development process has been criticized by companies from a number of perspectives. DoE should make the following changes:

1. Unfunded subtopics: Although the current process allows DoE to prioritize subtopics after proposals are received, this approach ignores the cost to applicants. DoE should seek to ensure that all published topics are funded. Non-funded topics should be a rare event—for example, when for unexpected reasons no fundable applications were received.
2. Recurring subtopics: Although subtopics rotate more often than topics, DoE should seek to ensure that the same subtopics do not recur regularly,

___________________

¹¹⁰ See Finding I-B and Finding I-D.

3. unless it has a specific reason to seek a technology that it has not yet found.

4. Noncommercial topics: DoE should enhance its cross-checks of subtopics to ensure that they meet DoE’s own definition of an appropriate internal rate of return for projects. It is not appropriate to write a subtopic for technology that is designed for use in a handful of National Laboratories and to then expect applicants to meet rigorous commercialization standards. More generally, DoE should clarify selection guidelines so that reviewers are clear on the appropriate balance between agency need and commercialization potential, which may vary by topic.
5. Because “open” subtopics provide funding opportunities for important projects that go beyond the areas defined in specified subtopics, they should be used for topics selected by EERE.

2. DoE should change the balance of funding to better reflect innovation and commercialization opportunities in the private sector.¹¹¹ In recent years, DoE has sharply limited opportunities in EERE, partly by eliminating open topics and partly by narrowing the published topics and subtopics. This action is a direct result of the way funds are allocated within DoE, with each program receiving more or less a pro rata amount of funding based on its overall extramural expenditures.

1. DoE should allocate funding based on the needs of the agency, not the individual divisions or programs.
2. Funding decisions should be data-driven, guiding funding toward opportunities that maximize the return to DoE’s investment (return on investment should be considered as a broad concept, not simply as revenue from commercialization).
3. DoE should consider reallocating funds in part based on a division or program’s willingness to work with SBIR/STTR to DoE should ensure that its solicitation is designed to meet all program objectives.

3. Although commercialization is an important program objective, DoE should not treat it as the only objective of the SBIR/STTR programs.¹¹²

1. DoE should ensure that a commercialization filter for applications is not only appropriate, but also used judiciously and does not result in insufficient levels of innovation or risk within the program. DoE will therefore need to develop mechanisms to identify and encourage high-risk/high-reward projects.

___________________

¹¹¹ See Finding I-B.

¹¹² See Finding I-B.

4. DoE should review and possibly rethink the relationship between the National Laboratories and SBIR/STTR.¹¹³

1. DoE should consider creating a task force to review the role of the National Laboratories in the SBIR/STTR program. Such a review is overdue and could lead to significant and positive improvements. As part of the review, DoE should seek to develop mechanisms for improving and strengthening the links between National Laboratories and SBIR/STTR. These could include:
   • Mandatory training in SBIR/STTR for DoE contracting officers with National Laboratories responsibilities
   • Incentives for utilizing SBIR/STTR technologies (e.g., prioritizing topics from National Laboratories staff with a track record in adopting such technologies)
2. Alternatively, DoE should consider ways to reduce linkages between the National Laboratories and SBCs, as the currently relationship may on balance be dysfunctional.

5. DoE should improve its application review system and monitor the profile of applicants.¹¹⁴

1. DoE should consider whether its current requirements for Phase I commercialization plans are appropriate.
   • DoE should consider whether a formal commercialization plan is appropriate for Phase I. There is evidence that early attention to commercial possibilities is important for eventual success, but many technologies find applications not envisaged accurately at early stages of development.
   • DoE should determine through post facto review whether Phase I commercialization plans have any correlation with commercialization activities after Phase II. If not, such plans simply add noise to the signal.
2. DoE should review the use of commercialization plans for Phase II.
   • DoE should undertake a detailed review to determine whether the Phase II plans submitted are useful predictors of eventual commercial outcomes.

___________________

¹¹³ See Finding VI-E.

¹¹⁴ See Finding I-C.

3. • DoE should consider whether a less burdensome set of requirements would provide sufficient information about company commercialization plans (which are in any event highly likely to change during Phase II).
4. DoE should ensure that the selection criteria are fully transparent.
   • If the final decision is based on DoE priorities, rather than application quality, then these priorities should be made public before the solicitation is published.
   • If however the final decision is based on application quality (rather than agency priorities), then DoE should consider adopting a scoring system that makes these differences transparent.
5. DoE should consider moving to a system in which reviewers arrive at a consensus score.
   • Such a system would require some one-to-one interaction between reviewers, which some companies observed would substantially improve the quality of reviews by making sure that the views of individual reviewers are challenged. Companies noted that NIH uses such a process, with success.
   • Despite the additional costs involved, DoE should consider adopting best practice from other agencies and competitions and add more reviewers. Two reviewers are simply too few to ensure that the review is fair.
6. DoE should find ways to allow companies to rebut poor-quality reviews and to elucidate their approach where necessary.
   • One option is for DoE to use available electronic tools to allow for a limited company rebuttal prior to selection.
   • Another option, which is used at NIH, is to permit companies to formally resubmit an application. This approach makes failures in the review process less definitive. DoE could consider adopting a formal resubmission process, although the single annual solicitation makes this a less promising alternative.
7. DoE should monitor the percentage of multiple awards and the composite age of company applicants (e.g., ratio of startups to mature companies) who are applying for and receiving awards.¹¹⁵

___________________

¹¹⁵ See Finding I-A-7.

6. DoE should further address the funding gap between Phase I and II awards.

1. DoE should permit companies to “work at their own risk.” Under this approach, used at the NIH SBIR program, companies can be paid for work completed during the gap period if they eventually receive a Phase II. This adds no risk to the agency and could shorten time to commercialization.
2. DoE should consider additional ways to provide financial support during funding gaps. For example, such support might be available to top scoring Phase II proposals as a supplement to their Phase I award, using an approach similar to that used at several DoD components.