1

Introduction

The Small Business Innovation Research (SBIR) program was conceived in the late 1970s and early 1980s to address several related, but distinct, challenges. The focus and purpose of the program, at least as articulated by Congress, has changed over time. The 1982 Act creating the program stated that the purposes were: to stimulate technological innovation, to use small business to meet federal research and development needs, to foster and encourage participation by minority and disadvantaged persons in technological innovation, and to increase private sector commercialization innovations derived from federal research and development (U.S. Congress, 1982). The 1992 reauthorization of the program also established the Small Business Technology Transfer (STTR) program, which sought to improve the commercialization of innovations resulting from federal funding by requiring collaborations between research institutions and small businesses. In addition, the 1992 legislation tweaked the purposes of the program slightly. The stated purposes of the program were changed to emphasize the program’s goal of increasing private sector commercialization of technology developed through federal research and development; to increase small business participation in federal research and development; and to improve the federal government’s dissemination of information concerning the program, particularly with regard to program participation by woman-owned small business concerns and by socially and economically disadvantaged small business concerns (U.S. Congress, 1992). The 1992 Act stated that the program had created jobs, but it was not until the 2000 reauthorization that legislative language was added to ask the National Academy of Sciences to evaluate the economic and non-economic benefits of the programs (U.S. Congress, 2000).

The principal budgeting mechanism of the SBIR program is a set-aside of each participating agency’s extramural federal research and development (R&D) budget. In fiscal year (FY) 2018, the SBIR set-aside for participating agencies was 3.2 percent of extramural R&D funding, and the STTR set-aside was 0.45 percent. Cumulatively, since its inception, the SBIR program had

provided over $43 billion in awards to small business by the end of 2016 (SBA, 2016b).

The SBIR program is one of the best known, most emulated, and extensively studied government R&D programs in the world. Countries as diverse as India, Taiwan, and the United Kingdom have adopted programs of a similar nature, sometimes using the same name in an attempt to more effectively engage small businesses in their national economies (BIRAC, n.d.; Liu, 2014; UK Government, 2020; Business.gov.nl, n.d.).¹ Horizon 2020 is a program within the European Union that funds innovative small and medium-sized businesses with a similar two-phase program (European Commission, n.d.). The SBIR program is the subject of numerous academic reviews, several studies by the Government Accounting Office, and two previous assessments by the National Academies of Sciences, Engineering, and Medicine, the second of which also reviewed the STTR program. However, despite the perceived effectiveness of the program, various stakeholders—academics, participants, administrators, and policy makers—have varying notions of the purpose and goals of the program. Even the congressionally stated purpose and goals of the program may themselves be in conflict. The challenge of assessing the program offers the opportunity for a fresh perspective on its contributions and effectiveness.

BACKGROUND AND CONTEXT FOR THE ESTABLISHMENT OF SBIR AND STTR

In the early 1970s, a number of government officials, led by Roland Tibbetts at the National Science Foundation (NSF), began seeking ways to increase technological innovation and commercial applications stemming from federally sponsored R&D. They expected that improving innovation and commercialization would stimulate economic growth, and they believed that small businesses were key in this endeavor. They held the notion that individuals and small businesses are “the primary source of category-creating inventions and technical breakthroughs” (Tibbetts, 2011). According to Tibbets, large firms do not pursue such innovation because they will not risk breaking from a so-far successful business model or product lines. Universities and government labs are not in the business of commercial development. However, unlike large firms, universities, and government labs, small businesses are specifically in business in the hopes of raising high-risk capital to develop new and innovative products with the potential to be major breakthroughs.²

Additionally, Tibbetts and others also believed that small businesses employ more scientists and engineers than any of the other organizational types.³

___________________

¹ Also see Inoue and Yamaguchi (2017), and Wang, Li, and Furman (2017).

² As discussed below, the committee’s review of the literature and DOE data on SBIR/STTR firm performance indicates that this trend is driven primarily by very young small businesses.

³ National Science Foundation data suggest that small businesses did not begin employing more than 10 percent of all full-time-equivalent scientists and engineers until 1984. Including those employed at academic or government institutions would decrease the percentage at small business even further. See the NSF’s Industrial Research and Development Information System table H-19 for employment data by firm size for years 1954, 1957-58, 1963-64, 1967-68, and 1971-99, and Keller and Block (2013), Table 1.

But despite these apparent advantages of small businesses for converting federal R&D investments into commercial innovations, then, as now, small businesses received the smallest share of funding by far: under five percent went to small businesses compared to 50 percent or more to large firms, around 35 percent to universities, and about 10 percent to non-profits, including national labs.⁴ Tibbets and others believed that small businesses’ participation was low because they could not compete well against large firms, universities, or national labs for awards in the standard federal R&D funding programs. Wanting to help small businesses compete for federal funding, and believing in their greater capability to foster innovation, the advocates began to push for a funding program that both would be available only to small business and would emphasize the commercialization of research results.

At the same time, these advocates also held the notion that universities and large businesses would not attempt to meet specific, important federal R&D needs. They believed universities shied away from such needs because the research required was too applied and thus too distant from core research interests. They also believed that large businesses would not pursue some research problems because the end market was too small, with just the federal government as the sole or primary purchaser. Tibbets and the others designed the SBIR program as a single solution to both of these two “paradoxes” of innovation funding. They hoped to provide a solution with twin purposes, first to provide “a transparent, competitive, and reliable source of early-stage funding for R&D based entirely on scientific merit,” and second, to ensure that “the government [could] obtain needed R&D that the private sector could not otherwise provide” (Tibbetts, 2011, p. 99).

Early program advocates saw this as an effort to allow small businesses to gain access to high-risk capital, and thus put a large set of scientists and engineers to work in service of innovation and national competitiveness at a larger capacity than in the absence of such a funding program. The timing coincided with national anxiety over competitiveness and employment as stagflation hit the economy. This staged the environment where Tibbets could suggest, and receive backing for, an experimental program at NSF that was open only to small businesses, and in 1977 NSF launched SBIR as an agency program (SBA, n.d.a).

A national interest in the role of small business in the economy and innovation had also grown around this same time in the late 1970s. In 1979 the Small Business Administration (SBA) had taken note of NSF’s SBIR program and began to work with Congress on expanding the program across the government. About the time of President Reagan’s first State of the Union Address, his administration produced the first in a series of reports for Congress

___________________

⁴ National Science Foundation, Survey of Federal Funds for Research and Development Fiscal Years 2017–18.

examining the economic role of small businesses across industries that detailed employment, production, and investment (SBA, 1982). These reports foreshadowed congressional actions that were designed to respond to the national economic anxiety and were specifically aimed at commercializing the outputs of federal R&D funding. Congress began passing a number of acts to this end, most during the Reagan administration, with a few during the George H. W. Bush’s administration (Block, 2008). Among the laws passed, the Small Business Innovation Development Act of 1982 created the SBIR program (U.S. Congress, 1982).

THEORETICAL SUPPORT FOR SBIR AND STTR

The longevity of SBIR, along with its sister program, STTR, and the size of its budget are due in large part to a strong theoretical justification for the program. The economics of knowledge and innovation suggest the need for a strong role for government in setting the long-term agenda for future scientific progress. Private firms, reacting to market forces, are not able to undertake long horizon investments for creative and disruptive technologies that advance social welfare.

Early support for this rationale dates back to Nelson (1959) and Arrow (1962), whose discussion of knowledge as a public good provided the dominant rationale for government support of innovative activity. Their work established that knowledge spillovers would create disincentives for potential private funders of innovation, who would be unable to fully realize the economic returns of their investment.

It has been shown that smaller and younger firms are more agile in adapting innovative capacities to shifting environmental contexts (Sørensen and Stuart, 2000) and that such firms generate higher rates of job growth (Neumark, Wall, and Zhang, 2011) than larger and more established firms. The dominant view until recently has been that it is the size of the firm that is important, and that firm size is negatively related to employment growth (Neumark, Wall, and Zhang, 2011). Haltiwanger, Jarmin, and Miranda (2013), however, provided strong empirical evidence that younger firms are the key drivers of high employment growth rates among small businesses and that the firm size does not predict employment growth once the age of the firm is considered. But these smaller and younger firms are at a competitive disadvantage in the marketplace given their much lower access to credit and capital required to survive the early stages of the innovation process, the so-called “valley of death” phenomenon (Auerswald and Branscomb, 2003). For firms that survive the valley of death, Haltiwanger, Jarmin, and Miranda (2013) found that younger firms outpaced their more mature counterparts in job creation.

The current literature lacks consensus around the optimal role of government in addressing private firms’ financial constraints, although several findings support the effectiveness of the basic structure and operations of SBIR and STTR. According to researchers, a major role for SBIR and STTR is to help

firms overcome credit and capital constraints at the early stages of the innovation process. Indeed, such “technology push” programs have been proven essential in the mix of policy instruments necessary to bring firms through the innovation process to commercialization, and this is especially true in the early stages of the process (Nemet, 2009). Beyond financial constraints, innovative firms benefit from collaboration with federal mission agencies, which define long-term goals to address pressing societal concerns. Further, technical innovations have been shown to be most successful when coupled with market opportunities (Arthur, 2007), a finding that provides strong justification for SBIR’s evaluation of innovative proposals that also demonstrate strong prospects for commercial success.

SBIR/STTR programs capitalize on gains from diversifying the government supplier base and promoting entry into technical fields. The programs open new procurement pipelines for federal agencies while allowing the agencies to support small business innovations that advance their missions. Products emerging from these symbiotic relationships can have dramatic commercial impacts. Sherwin and Isenson’s (1967) early work on the subject demonstrated hundreds of innovations that came from government-funded weapons systems. More recently, Mazzucato (2013) illustrated the same procurement-commercialization link by pointing to government funding for what became numerous components of Apple’s iPhone.

CONFLICTING PROGRAMMATIC GOALS

The SBIR/STTR programs, and any study of them, face a major challenge. The various constituent groups in the programs’ community espouse different views of the programs’ purpose and goals, and this results in differing expectations of what the programs should accomplish, what they should provide to awardees, how they should be implemented, and what outcomes should be expected. A full program assessment requires a clear statement of program goals and expected outcomes to use for measuring progress or success.

Solving the dilemma is not as easy as looking at purpose and goals statements from the federal government, because there are many versions coming from various federal actors. Congressional statements differ from those from the SBA, for example. Further, Congress’s legislative request for this study asks that the study try to measure certain outcomes, but some of those do not appear to stem from the goals enunciated in the Small Business Act of 1982 that established the SBIR program. Looking at the statements of industry groups or at the goals assessed by researchers only complicates the matter.

Complicating matters even more, the statements of goals appear to be in conflict. Three of the goals listed on the sbir.gov website, for example, include: (1) stimulate technological innovation”; (2) “meet federal research and development needs”; and (3) “increase private-sector commercialization of innovations derived from federal research and development funding” (SBA, n.d.b). A well-known challenge of innovation processes, however, is the gap

between research and commercialization. Individuals skilled at research tend to have much lower capability for translating their research into products and then commercializing them, and vice versa (Toole and Czarnitzki, 2007). Many expressions of the program’s goals emphasize commercialization, which could lead to a funding prioritization of projects that promise short-term commercialization potential over those with potential for longer-term innovation potential. DOE faces a particular challenge regarding commercialization since, in large part, it does not have a procurement stream for eventual products that may arise from research that it funds. Rather, eventual products would be targeted at private markets. Consequently, a push for commercialization may lead DOE to favor projects that could produce highly specialized tools or equipment for the agency’s national laboratories, but with limited larger commercial potential.

The fourth goal stated on sbir.gov is for agencies to “foster and encourage participation in innovation and entrepreneurship by women and socially or economically disadvantaged persons” (SBA, n.d.b). Therefore, agencies administering the SBIR program must try to balance their agency missions and goals with programmatic goals where any, and sometimes all, goals are in tension if not conflict with one another. Small businesses that participate face a very similar challenge. Essentially the program asks that agencies and awardees solve research problems, solve commercialization problems, and diversify participation at the same time as a means to address the overall societal mission of their agencies. The goals of STTR also come into conflict with each other and create similar challenges.

THE NATIONAL ACADEMIES STUDY

Congress first requested that the National Academies undertake a study of the SBIR program as part of the Small Business Reauthorization Act of 2000. This study mandate was expanded in the National Defense Authorization Act for Fiscal Year 2012, wherein Congress directed agencies with SBIR program budgets of more than $50 million to engage with the National Academies to conduct a quadrennial assessment of each agency’s SBIR and STTR programs.⁵

The congressional mandate calls for assessments to study “how the SBIR program has stimulated technological innovation and used small businesses to meet federal research and development needs” and to include several specific analyses and evaluations. The first objective is to assess the value of the R&D conducted under the program and evaluate its quality. The second objective is to evaluate the economic and benefits with rate of return, and compare that to other federal R&D expenditures. The third objective is to evaluate non-economic

___________________

⁵ Two previous sets of studies have been completed by the National Academies in response to the legislative mandate. The first study was completed in 2009 and included a study of the SBIR programs at the Department of Defense, National Institutes of Health, DOE, National Aeronautics and Space Administration, and NSF. The second was completed in 2016 included studies of both SBIR and STTR at the same agencies.

benefits. The mandate also asks for an analysis of whether federal agencies are making sufficient effort to utilize Phase II participants to fulfill procurement needs. Since 2011, the legislative mandate also calls for a study of how the STTR program has “stimulated technological innovation and technology transfer” (15 USC 638 note, 2019.)

The purpose of this set of National Academies studies is to undertake a detailed examination of the economic and non-economic benefits the SBIR and STTR programs at DOE in compliance with the legislative mandate. Drawing on published research plus existing data, the committee has examined (1) a range of economic impacts including, to the extent practicable, the number of jobs created by these programs; (2) the role of SBIR/STTR programs in stimulating technological innovation and contributing to DOE’s research and development needs; (3) collaborations created between small businesses and research institutions on account of the programs;⁶ (4) the effectiveness of DOE’s SBIR/STTR award-selection process and commercialization assistance; and (5) ways to improve outreach efforts to SBIR and STTR applicants, particularly to increase applications from small businesses that are new to the programs, from underrepresented states, woman-owned, or minority-owned. As part of these activities, the committee has also examined multiple award recipients and their implications for program effectiveness. The committee’s formal statement of task is included in Box 1-1.

THE SBIR PROGRAM: A HISTORY

The SBIR and STTR programs are codified in Section 9 of the Small Business Act, 15 U.S.C. Section 638. The legislation authorizing the programs has, since the beginning, contained sunset provisions, and the programs are currently authorized through 2022.

This legislative and policy background was instantiated in a series of concrete aims in legislation, beginning with the Small Business Innovation Development Act of 1982, which amended the Small Business Act “to strengthen the role of small, innovative firms in federally funded research and development, and to utilize federal research as a base for technological innovation to meet agency needs and to contribute to the growth and strength of the nation’s economy” (P.L. 97-219).Currently the SBIR website states that the mission of the SBIR program is “to support scientific excellence and technological innovation through the investment of federal research funds in critical American priorities to build a strong national economy” and lists the programs goals as four-fold:

___________________

⁶ The unique feature of the STTR program is the requirement for the small business to formally collaborate with a research institution in Phase I and Phase II.

• Stimulate technological innovation
• Meet federal research and development needs
• Foster and encourage participation in innovation and entrepreneurship by women and socially or economically disadvantaged persons
• Increase private-sector commercialization of innovations derived from federal research and development funding (SBA, n.d.b)

Similarly, the STTR program, created in 1992, has nearly identical goals, especially with regard to commercializing federal R&D outputs. The STTR website states that “the mission of the STTR program is to support scientific excellence and technological innovation through the investment of federal research funds in critical American priorities to build a strong national economy” and lists the three programs goals as to

• stimulate technological innovation,
• foster technology transfer through cooperative R&D between small businesses and research institutions, and
• increase private sector commercialization of innovations derived from federal R&D (SBA, n.d.c).

While the specific language from SBA detailing SBIR’s mission and goals has naturally evolved since 1982, it still evinces a desire to stimulate innovation, address federal R&D needs, involve small businesses in federal R&D support, commercialize federal R&D outputs, and make participation in innovation more accessible for women and other disadvantaged persons. The focus on underrepresented minorities was present among the SBIR program goals from the beginning.⁷

At the outset, the legislation governing the SBIR program called for participation by any federal agency with extramural research or R&D budgets in excess of $100,000,000. The STTR program has set a higher threshold, requiring participation by any agency with research or R&D budgets in excess of $1,000,000,000. As agency budgets increase, new participants may, and have, come in.

The latest program reauthorization took place in 2016 as part of the FY 2017 National Defense Authorization Act, which extended the programs until 2022. More recently, the FY 2019 National Defense Authorization Act extended some of the pilot programs through FY 2022 and added additional commercialization assistance. The programs have experienced changes over time, principally when reauthorized. Notably, the 1992 reauthorization added language to highlight that commercial potential needed to be considered. One of the purposes of the SBIR program was changed “to emphasize the program’s goal of increasing private sector commercialization of technology developed through federal research and development” (U.S. Congress, 1992). The 1992 legislation also explicitly added support for woman-owned businesses by changing one of the program’s purposes from “to foster and encourage participation by minority and disadvantaged persons in technological innovation” (U.S. Congress, 1982) to “to improve the federal government’s dissemination of information concerning the small business innovation research program, particularly with regard to program participation by women-owned small business concerns and by socially and economically disadvantaged small business concerns” (U.S. Congress, 1992). The 2000 reauthorization also added additional language around commercialization and specifically mentioned that commercial potential should be used as a criterion for awards. Specifically, the Small Business Act was modified “to provide for the requirement of a succinct commercialization plan with each application for a second phase award that is moving toward commercialization” (U.S. Congress, 2000). The 2000 reauthorization also initiated the National Academies’ assessments of the program. This assessment mandate was expanded in the reauthorization of the programs in the National Defense Authorization Act for Fiscal Year 2012, wherein Congress directed

___________________

⁷ The Small Business Innovation Development Act of 1982 listed four goals in the establishment of the SBIR program: “to stimulate technological innovation; to use small business to meet federal research and development needs; to foster and encourage participation by minority and disadvantaged persons in technological innovation; and to increase private sector commercialization of research derived from federal research and development” (U.S. Congress, 1982).

agencies with SBIR program budgets of more than $50 million to engage with the National Academies to conduct a quadrennial assessment of each agency’s SBIR and STTR programs.

Over time, the SBIR and STTR programs have enjoyed considerable support within Congress and various administrations, largely on a bipartisan basis. The percent set-aside for each program has increased over time. For FY 1983, the percentage to be set aside for the SBIR program, based on the original legislation, was no less than 0.2 percent of a participating agency’s extramural budget, with this percentage increasing over time to 1.25 percent. When the STTR program was established, it required a set-aside of at least 0.05 percent for the program in FY 1994, a rate that was prescribed to increase to 0.15 percent. Subsequent legislation increased these percentages; the FY 2011 Reauthorization increased the percentage of each over this decade, finally leading to today’s minimum rates of 3.2 percent for SBIR and 0.45 percent for STTR. Combined with increasing agency extramural R&D budgets, the result has been a significant expansion of the programs.

Agencies enjoy wide latitude in the operation of the SBIR and STTR programs. Ultimately the legislation governs the programs and provides the criteria according to which program managers are accountable. Research and previous assessments of the programs have moved away from the multidimensional and agency-focused nature of the programs’ legislative goals to focus on other concerns. These studies have introduced new considerations and created confusion about the programs’ goals. In this assessment, the committee’s considerations are limited to the legislative mandate.

PROGRAM OPERATION

SBIR and STTR program operations are decentralized to agencies and sub-agencies throughout the federal government,⁸ with the SBA playing a broad oversight role. Though there is significant variation across and within agencies—and agencies have engaged in adaptation and experimentation in their programs—a number of key features of the programs stand out, and the broad structure of SBIR and STTR is similar across agencies.

A Three-Phase Program

By design in the original Small Business Act, the SBIR program proceeds in three phases:

___________________

⁸ Eleven federal agencies currently operate SBIR programs. In addition to DOE, these include the departments of Agriculture, Defense, Education, Health and Human Services, Homeland Security, and Transportation; the Environmental Protection Agency; National Aeronautics and Space Administration; NSF; and the National Institute of Standards and Technology and National Oceanic and Atmospheric Administration at the Department of Commerce. STTR programs are operated at the Department of Defense, Department of Health and Human Services, National Aeronautics and Space Administration, DOE, and NSF.

• Phase I: This phase is a feasibility demonstration phase intended to support projects that seek to “establish the technical merit, feasibility, and commercial potential” of a line of research or R&D. It also provides opportunity and information for the agency to assess the ability of the small business to conduct the work prior to providing any further federal support, especially through a Phase II award. At the time of writing, DOE awards were set at a maximum of $250,000 over six to 12 months.⁹
• Phase II: This phase is meant to support Phase I projects that showed positive results and that show continued scientific and technical merit, along with commercial potential. After a trial of allowing applications that have already proven feasibility, only Phase I awardees are eligible to apply for Phase II funding. At the time of writing, DOE awards did not exceed $1,600,000 over two years.
• Phase III: This phase receives no funding from the SBIR program. Instead, it refers to follow-on funding in pursuit of commercialization, which may include the direct purchase of the product. Congress’s original intent was for this phase to be where “non-federal capital pursues commercial applications of the research or research and development” (U.S. Congress, 1982) or where non-SBIR federal follow-on funds support additional research or “production contracts for products, processes or services intended for use by the U.S. government” (SBA, n.d.b).

STTR consists of the same phases and dollar amounts, but small businesses receiving STTR awards are required to formally collaborate with a research institution (such as a university or federal lab) in Phase I and Phase II.

In addition to standard Phase I and Phase II awards, some agencies may provide additional funding either prior to a Phase I or to follow a Phase II award. Pre-Phase I assistance, so-called Phase 0 awards, are sometimes given to assist potential applicants in completing a Phase I application. At DOE these are targeted to first-time applicants. Follow-on Phase II awards, known as Phase IIA and Phase IIB, may be given to awardees with promising technologies that could not complete Phase II on time or otherwise are not yet ready for commercialization. At DOE, such continuation awards consist of up to $1.1 million over two years. In addition, in the case of SBIR, firms that received a Phase I, Phase II, and either a Phase IIA or IIB from DOE are also eligible to apply for an additional award to aid in commercialization, called a Phase IIC award that is capped at $1.1 million. DOE implemented Phase IIC awards after the FY 2019 National Defense Authorization Act mandated that agencies implement a commercialization assistance pilot project (U.S. Congress, 2019).

___________________

⁹ Agencies can request and receive approval from SBA to exceed Phase I and Phase II award amounts. Additional funds are available for commercialization assistance.

A Competitive Program

SBIR and STTR awards are made on a competitive basis, with each participating agency issuing solicitations—or funding opportunity announcements—at least once per year with topics that it develops to reflect agency priorities. Phase I proposals are evaluated on the basis of “scientific and technical merit and feasibility,” with consideration of potential commercialization. Agencies may fund several proposals in response to a funding opportunity announcement. Receipt of Phase II funding is based on Phase I results and an evaluation of the “scientific and technical merit, feasibility, and commercial potential of the Phase II proposal” and are awarded at the discretion of the agency. Commercial potential “includes the potential to transition the technology to private sector applications, government applications, or government contractor applications.” Applicants who have won multiple previous awards must meet agency benchmarks for commercialization of prior SBIR and STTR awards in order to receive a new Phase I award (SBA, 2019).

Eligibility Rules

To receive an SBIR Phase I or Phase II award, an awardee must qualify as a small business concern as defined in SBA’s regulations at 13 C.F.R. Sections 121.701–121.705, and the primary employment of the principal investigator must be with the small business concern during the period of award. At least two-thirds of Phase I and one-half of Phase II SBIR effort must be performed by the awardee, with some room for exceptions.

Under STTR, with its requirement that awardees formally collaborate with a research institution (such as a university or federal lab) in Phase I and Phase II, the rules are somewhat different. Awardees must qualify as a small business concern just as under SBIR, but the rules state that for both Phase I and Phase II, no less than 40 percent of the R&D must be performed by the small business, and no less than 30 percent of the R&D must be performed the partnering institution. The primary employment of the principal investigator must be with either the small business or the research institution during the period of award.

As a rule, the Phase I and Phase II research and R&D effort under each program must be completed in the United States. Companies that are majority-owned by multiple venture capital companies, hedge funds, or private equity firms are permitted but are restricted to no more than 25 percent of the agency’s SBIR funds for the National Institutes of Health, DOE, and NSF and no more than 15 percent of agency SBIR funds for all other agencies.

The Application and Award Process

For each program, the participating agency issues solicitations—or funding opportunity announcements—at least one time per year with topics it

develops to reflect agency priorities.¹⁰ In view of the program goals to “encourage participation in innovation and entrepreneurship by women and socially or economically disadvantaged persons” (SBA, n.d.b), agencies engage in outreach activities to encourage applications from underrepresented groups. Each solicitation topic broadly outlines areas of research that support agency priorities and may have several subtopics designed to support specific aspects of the broader topic. Solicitations include eligibility requirements, application requirements, review criteria, and due dates, and as part of their applications, prospective awardees must specify how the proposed research will respond to a particular topic and subtopic and meet other requirements described in the solicitation.

SBA encourages agencies to use their “routine review processes” for SBIR and STTR awards, whether they be internal or external. The peer review process combines elements of traditional peer review, assessment of commercial potential, and program manager discretion, with Phase I applications evaluated on the basis of “scientific and technical merit and feasibility” and commercialization considerations and the receipt Phase II awards based on Phase I results and the “scientific and technical merit, feasibility, and commercial potential of the Phase II proposal” (SBA, 2019).

Awards are made either as contracts or grants, depending on the agency. Most agencies are required to issue awards no more than 90 calendar days after the close of a solicitation.¹¹ Awardees are required to submit certification information at various points during their Phase I and Phase II awards and are required to submit commercialization information at the end of a Phase II award (and are asked to do so voluntarily on an annual basis for at least five more years). Once a grant is made, there is not typically a high level of engagement between an agency and an awardee. Awardees’ work is largely “hands-off” and not characterized by the high level of engagement of, for example, cooperative agreements used at the Advanced Research Projects Agency-Energy or the direct publish-or-perish cycle of academia. SBA has initiated a database to track awards, and it maintains some data on applicants including a section that allows companies to report on commercialization results for those awards. However, with regard to measuring outcomes at DOE, because individual offices’ missions inform their SBIR/STTR goals and vary from office to office, there is less systematic agreement as to the appropriate metrics for technical and innovation outcomes. There is currently no centralized, systematic database that allows mapping from the grants into long-term outcomes. Agencies are, however, required to maintain a database of certain information on awardees and to provide an annual report to SBA and the White House Office of Science and Technology Policy. As of this writing, the current authority requires that annual agency performance plans to Congress include sections on SBIR and STTR.

___________________

¹⁰ The topic development process is described in greater detail in Chapter 3.

¹¹ In order to reduce cycle times and resulting funding gaps between phases, some agencies employ “fast track” awards in which Phase I and Phase II application information is reviewed in one step.

Tailoring SBIR/STTR to DOE

DOE has tailored its implementation of the SBIR/STTR programs, as have other administering agencies, to support the agency’s overall mission, along with the specific missions and goals of its programmatic offices and research programs that fund awards.¹² This can create instances of conflict between agency mission and the program goal of commercialization by small businesses, and a conflict between the programs’ goals of innovation and commercialization. DOE can tailor its implementation processes because, while Congress charged the SBA administrator with overseeing and coordinating the program activities of participating agencies, it also granted each agency latitude for determining how it will operate its SBIR and STTR programs. Specifically, each has power to determine the categories of projects and solicitation topics, issue solicitations, receive and evaluate its own proposals, make final award decisions, and make and manage its own funding agreements.

Each agency has followed this authority to develop its own set of processes for conducting its programs. DOE has established processes that aim to support its own mission and the missions of its programmatic offices and research programs that sponsor SBIR/STTR activities.¹³ DOE has centralized some functions and activities into an agency-wide SBIR/STTR Programs Office housed within the Office of Science while assigning others to its programmatic offices and research programs. Within the agency, six program offices across DOE and six research programs housed in the Office of Science participate in the programs.¹⁴ Each participating office or division has authority or obligation to develop topics for solicitations and suggest individuals who could serve as application reviewers. DOE is unique compared to other federal agencies in that it has a large number of a highly specialized and geographically dispersed federally funded research and development centers. The national energy labs are a critical part of the U.S. innovation infrastructure with different areas of technical expertise and distinct missions (NSF, 2019).

While the processes used across all the offices share some similarities, they also show diversity reflective of the flexibility for participating agencies to use the program and small business participants to meet the offices’ own missions and goals. When developing solicitation topics, for example, some offices have uniform policies while others allow staff to tailor the process to best suit the needs of the program they manage. In general, all offices rely heavily on research and

___________________

¹² While specific research and development needs evolve with market conditions, the overall missions of the DOE program offices are listed in Table 3-1.

¹³Chapter 3 more fully describes and examines the processes employed to operate the programs at DOE.

¹⁴ There are currently 13 program offices that participate in SBIR/STTR programs. After the committee completed its interviews, the Office of Electricity Delivery and Energy Reliability was divided into the Office of Electricity and the Office of Cybersecurity, Energy Security and Emergency Response. See https://www.energy.gov/science/sbir/small-business-innovation-research-and-small-business-technology-transfer.

industry stakeholders to suggest what research or programmatic needs the topics should address. Similarly, to identify potential peer reviewers to assess applications, some program offices search only from a linked community and some rely on the staff who develop topics, while others engage in broader outreach. While relying on stakeholders can clearly support agency goals, these processes may not support the overall SBIR/STTR program goals, and can be potential sources of conflict.

Approaches to award selection show an emphasis on supporting individual offices’ missions rather than emphasizing other SBIR/STTR program goals such as commercialization or increasing participant diversity. Each office contacted in the committee’s analysis reported that its processes are tailored first to the office’s or program’s mission, along with other factors such as internal expertise and capacity. Offices with fewer budget control lines from Congress indicated that this is simpler and more easily put into practice than programs with more control lines. The committee’s conversations with program offices also suggest that they emphasize their technical program missions ahead of SBIR/STTR program goals. For instance, when reviewing applications, essentially all offices stated in some way or another that they instruct reviewers first to assess the technical merit. A minority of offices stated that commercialization is also very important, while at least one noted that its mission is so early in the cycle of technology development that commercial potential often is difficult to see in proposals that help fulfill that office’s program mission. At least one noted that it tries to select projects that look ready to commercialize in less than a year; too great an emphasis on near-term commercialization almost by definition precludes supporting long-term innovation, a major SBIR/STTR program goal. Effectively, then, the offices find themselves forced to make tradeoffs between selecting projects that support one of the SBIR/STTR program goals, but that do not support one or more of the others.

The offices also expressed difficulty attempting to increase demographic diversity of awardees since they believe that the system used to manage applications does not provide demographic details on applicants. Similarly, several offices explained that while they seek subject matter expertise when recruiting possible reviewers, they do not seek demographic diversity. Only one office, Energy Efficiency and Renewable Energy, indicated that it takes specific steps to diversify its reviewers along demographic lines while also emphasizing that technical merit, not demographics, is the key driver of award selection. The committee was unable to gain access to application information that would be necessary to analyze the extent to which program administration may affect the review and selection of underrepresented groups.

Chapter 3 provides more detail about differences across program offices and provides findings and recommendations about potential changes.

THE CHALLENGE OF PROGRAM EVALUATION

Evaluating any program that supports risky, early-stage technologies carries significant challenges. The development of such technologies involves long and uneven time lags, and their impacts are often far removed from what was originally envisaged. Ultimate “home runs” from SBIR- or STTR-funded research may occur in domains far distant from the initial firm or domain of funding that was initially supported. However, as a reasonable, though highly imperfect rule, evaluators should be able to identify some long-term impact grounded either in companies, technologies, or people that have been influenced by the portfolio of investments from the DOE SBIR/STTR program and have made a significant economic or technical contribution. There should also be medium-term metrics that show meaningful progress toward economic and technical goals that will be necessary first steps or surrogate markers for achieving more impactful outcomes over time. Moreover, both these medium-term markers and long-term outcomes should reflect effective operational and policy aspects of the programs that allow them to achieve their (multiple) objectives in a cost-effective, proactive, and inclusive manner. That said, there are many central challenges of evaluating an innovation assistance program:

• A key question of evaluation is not whether the SBIR and STTR programs happened to fund particular companies or technologies that have been successful and impactful, but rather whether the programs and participation in the programs meaningfully improved or enhanced those outcomes. In other words, did the program provide “additionality”? To evaluate this requires not simply assessing whether SBIR/STTR firms or projects have done well on average, it means assessing how well they would have done in the absence of the program. While this is very difficult to actually measure (though can be achieved in some cases through a control group), it is nonetheless an important criterion to keep in mind when considering the analyses that follow.
• Outcomes depend critically on implementation and the processes by which the programs are implemented. This requires detailed attention to the ways in which the programs define the topics for solicitations, review proposals, and interact with awardees. Job creation may be more indirect than direct because the SBIR/STTR program by design encourages awardees to form external R&D partnerships, rather than increase their own staff.
• Some of the outcomes, perhaps even some of the most important ones, will be outside the domain of a systematic study. For example, there are not reliable data to ascertain whether product introductions occurred, and those that occurred may have been for highly specialized technologies and equipment with few buyers worldwide.

• Obtaining evidence on outcomes is challenging due to company name changes and the long time lags involved in bringing research to market.
• Failure is inherently a part of the innovation process.
• There is a need to balance SBIR/STTR objectives with mission objectives of each federal agency.
• Ultimately, there is no single lens with which to evaluate the impact of SBIR/STTR, and most persuasive forms of evidence will emerge from multiple signals and multiple analytical approaches. Multiple reinforcing markers of overall impact or process on patterns of impact and goal achievement are where we find the most persuasive evidence for our cross-cutting findings and recommendations.

As already noted, the committee was unable to gain access to SBIR/STTR application information. It was also unable to gain access to partnering data for SBIR awardees. Application data are necessary to analyze the extent to which program administration may affect the review and selection of underrepresented groups and would permit such things as a deeper analysis of program additionality through a comparison of successful versus unsuccessful applications. Partnering data for SBIR awardees are necessary in order to gain greater understanding of the program collaborations with academic and federal research organizations and the degree to which these connections may vary for different types of awardees. Future analyses would benefit greatly from access to these data.

OUTLINE OF THIS REPORT

The remainder of the report contains detailed background on the programs and from the literature, describes the committee’s methodology and results, and gives the committee’s findings and recommendations.

Chapter 2 presents a complete review of the literature on SBIR and STTR along with relevant literature on energy innovation and innovation generally. It describes the policy rationale for the programs and reviews academic research plus broader publications such as government reports on and assessments of the programs, as well as earlier National Academies reports.

A description of the organization and administration of the SBIR and STTR programs at DOE follows in Chapter 3. Chapter 4 describes the landscape of SBIR/STTR winners since 2005 and provides information about the location of firms winning SBIR and STTR grants relative to other firms and clusters of small business and innovative activity. This chapter also examines differences in these trends for several relevant sub-groups of awardees: woman-owned firms, underrepresented minority–owned firms, first-time SBIR/STTR winners, and multiple award recipients. Chapter 5 focuses on innovation and scientific outcomes. The body of the report is followed by a list of references, agendas for meetings of the committee, biographies of committee members, and a technical appendix.

This page intentionally left blank.