National Academies Press: OpenBook

SBIR/STTR at the Department of Energy (2016)

Chapter: 7 Insights from Case Studies and Survey Responses

« Previous: 6 Participation of Women and Minorities
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

7

Insights from Case Studies and Survey Responses

This chapter reviews a range of impacts of the Small Business Innovation Research (SBIR) and Small Business Technology Transfer (STTR) programs based on written responses to open-ended questions solicited in the 2014 Survey and interviews with executives for the case studies. The survey process is described in Appendix A, and the survey instrument is provided in Appendix C. Data from the survey are used to support analysis throughout the report; this chapter draws from the written, open-ended responses to survey questions. These responses can be divided into views about the concept and advantages of the SBIR and STTR programs, where the comments were generally positive, and those on the operation of the program, where the comments were often more critical. Box 7-1 lists the case study firms, all of which were DoE SBIR and/or STTR award winners. Full case studies, carried out in 2015-2016, can be found in Appendix E.

A wide range of companies were selected for case studies. They varied in size from fewer than 5 to more than 100 employees. They operated in a wide range of technical disciplines and industrial sectors. Some firms focused solely on serving the national labs, while others focused on commercialization through the private sector. Overall, this portfolio sought to capture many of the types of companies that participate in the SBIR/STTR programs.

The 12 case study participants all represent Phase I and Phase II SBIR or STTR winners. Among these, ony two—Diversified Technologies and Woodruff Scientific—have not received STTR awards. Some, most notably Creare and Physical Sciences, have won many awards over a number of years. Creare has won 959 SBIR/STTR awards since 1985 and Physical Sciences 1,108 SBIR/STTR awards since 1983. Six companies have won fewer than 100 awards each, the fewest represented by Woodruff Scientific, which has won 3 SBIR Phase I and 3 SBIR Phase II awards. Two case studies covered

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

firms that were profiled in the 2009 National Research Council1 assessment of the DoE SBIR Program (See Box 7-2).2

Companies selected for case studies are not intended to be statistically representative of DoE SBIR/STTR award winners or their award outcomes. Although the number of case studies completed as part of this study is limited, case studies of selected firms can offer qualitative evidence about experiences with the program of firms that have achieved some success and may have acquired some insights regarding how the SBIR/STTR programs, particular aspects of the programs, or the manner in which the company utilized the programs may have contributed to that success. Interviewees were also asked to raise any problems and provide their own recommendations about how the programs could be improved. Future research could benefit from a broader base of case study companies, including less successful companies, whose responses could provide a useful comparison.

This qualitative review provides needed context for the data discussed in Chapter 5 and aids understanding of the perspectives of award recipients as well as those who did not receive Phase II funding for what they considered to be a highly promising project.

___________________

1 Effective July 1, 2015, the institution is called the National Academies of Sciences, Engineering, and Medicine. References in this report to the National Research Council, or NRC, are used in an historic context identifying programs prior to July 1, 2015.

2National Research Council, An Assessment of the SBIR Program at the Department of Energy, Washington, DC: The National Academies Press, 2009.

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

The chapter summarizes case study and survey data regarding program impacts and management issues. Views summarized do not necessarily reflect the views of the committee. The chapter is organized into three main sections:

  1. Outcomes and Company Impacts
  2. Program Management
  3. STTR

The material in this chapter provides the first wide-ranging publicly available feedback of the DoE SBIR/STTR programs from program recipients.

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

OUTCOMES

This section focuses on outcomes related to commercialization and knowledge effects and on the ways in which SBIR/STTR supports the growth of an expanding ecosystem of innovative companies in energy-related fields.

In general, the case studies support the view that the programs are meeting their Congressional objectives except for encouraging the participation of women and minorities.

Commercialization

Discussions with company executives suggest that the perspective on commercialization has recently and substantially changed, for companies and for DoE itself. Dr. Rozzi (Creare) observed that in the 1980s the DoE SBIR/STTR programs primarily focused on research. Topic Managers (TMs) supported favorite technology projects, generally without a clear path to transition, which usually resulted in little commercial return. Company executives noted, and DoE staff confirmed, that topics have become more commercially oriented within the past 10 years. More recently, DoE has extended its requirements for Phase I and Phase III applications, placing more pressure on companies to have a well-defined commercialization plan in place, and companies have begun to view SBIR/STTR through a more commercial lens.

However, as discussed in Chapter 2, the DoE topic development process still does not align tightly with the drive for improved commercial outcomes, and significant disconnects exist between the SBIR program and downstream opportunities that might be available within DoE.

Commercial Results

Progress toward commercialization is primarily analyzed in Chapter 5, which focuses on results from the 2014 Survey. However, even though the information about private companies is limited, it is apparent that some DoE SBIR/STTR awardees are generating substantial income from Phase III and beyond. For example, according to the Hoovers/Dun & Bradstreet database, LI-COR generates more than $100 million annually, with an additional $14.8 million from its pair of European subsidiaries.3 Dr. Rozzi noted that nearly 40 percent of Creare’s total revenues are derived from Phase

___________________

3 “Li-Cor, Inc. Revenue and Financial Data,” http://www.hoovers.com/companyinformation/cs/company-profile.Li-Cor_Inc.b4245b76644713ab.html; “LI-COR BIOSCIENCES UK LTD Revenue and Financial Data,” http://www.hoovers.com/companyinformation/cs/revenue-financial.LI-COR_BIOSCIENCES_UK_LTD.650c82d750559ca4.html; “LI-COR Biosciences GmbH Revenue and Financial Data,” http://www.hoovers.com/companyinformation/cs/revenue-financial.LI-COR_Biosciences_GmbH.69ff7789c35f7144.html. Hoover does not report a year; revenue is assumed for 2014.

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

III commercialization activities related to past SBIR/STTR projects. Other case study companies report substantial commercial results. However, neither through the survey nor the case studies was the committee able to identify a “home run”: LI-COR is the most commercially successful DoE SBIR/STTR company reviewed for this assessment. Program outcomes are addressed in more detail in Chapter 5.

Spinouts

Some of the case study companies (notably Creare and Physical Sciences [PSI]) use spinouts as a primary mechanism for commercializing technology. Creare has spun out 10 companies in its history. These include the leading suppliers of plasma-based metal-cutting systems, Hypertherm, and of computational fluid dynamics software, Fluent, which was acquired for a substantial price by ANSYS in 2006. Although Creare remains a small company, these companies have generated more than 2,000 jobs and $500 million in revenues annually, according to Creare.4

The most recent Creare spinout is Edare, which provides manufacturing and product development services intended to transition innovative technologies into low- and medium-volume production. The Edare model centers on having two or three programs in production at any one time, providing low- to medium-volume manufacturing for government clients (although some commercial clients are anticipated). This low-volume production may be the end of the transition path for some products, but may also be an important way station on the path to larger volume sales or a licensing agreement once the technology is fully developed and the manufacturing processes rolled out. Dr. Rozzi observed that it is a good model for producing 30 to 50 units, which is a difficult level to achieve in a research and development (R&D) environment. (See Table 7-1.)

Physical Sciences, Inc. (PSI) subsidiaries tend to replicate the R&D culture of the parent company (many publications in peer-reviewed journals, use of SBIR funding), to focus on a limited (but stable) commercial opportunity, and to perform prototyping and low volume manufacturing.

In addition to establishing subsidiaries, PSI has spun out technologies into new companies. According to Dr. Green, the CEO of PSI, spinouts typically depend on venture backing and follow business models that target larger commercial markets with associated needs for product development,

___________________

4 “Cryogenic Machining Technology,” http://www.gearsolutions.com/news/detail/7168/cryogenic-machining-technology-from-mag; Jay Rozzi, “Cryogenic Machining Background and Application to Shipbuilding,” NSRP All Panel Meeting, October 2011, http://www.nsrp.org/6-Presentations/Joint/100411_Cryogenic_Machining_Background_and_Application_to_Shipbuilding_Rozzi.pdf, p. 4.

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

TABLE 7-1 Creare Spinouts

Company Year Spun Out
Hypertherm 1968 Hypertherm was founded to commercialize plasma cutting technology developed at Creare. Still headquartered in New Hampshire, Hypertherm is now the world’s largest manufacturer of plasma cutting tools.
Creonics 1982 Creonics develops and manufactures motion control systems for industrial processes. Acquired by Allen-Bradley in 1990, Creonics is now part of Rockwell International.
Spectra 1984 Spectra is a manufacturer of high speed ink jet print heads and ink deposition systems. Formed around a sophisticated deposition technology developed at Creare, Spectra was acquired by Fujifilm in 2006 and renamed Fujifilm Dimatix.
Fluent 1988 Based on Creare’s longstanding expertise in computational fluid dynamics, Fluent began marketing comprehensive computational fluid dynamics software. In 2006 ANSYS Inc. acquired Fluent for $565 million.
Mikros 1991 Based on Creare’s advanced electric discharge machining technology, Mikros offers precision micro-machining services.
Verax Biomedical 1999 Verax was founded to commercialize technology to detect bacterial contamination of cells and tissues intended for transfusion and transplantation. They have received seven rounds totaling $28.2 million in venture funding.
Edare 2011 Edare provides manufacturing and product development services intended to transition innovative technologies into low- and medium-volume production.

SOURCE: Creare.

manufacturing, logistics, and sales and marketing. Typically, these technologies have presented the opportunity for selling products to mass markets. PSI may take an equity stake in the company, but most of the funding comes from the venture community.

Dr. Green said that intellectual property (IP) and staff usually go with the spin-out. None of the spinouts has been highly successful, and many of the staff have returned to PSI. One spinout still exists but has been sold three times.

Knowledge Effects

The DoE SBIR/STTR programs have supported the development of numerous innovative technologies. Each of the case study companies has developed technologies that introduced new capabilities to the marketplace. Some companies have developed important, industry-leading technologies. For example:

  • LI-COR’s methane monitoring tools are a global leader, with more than 30,000 units sold.
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
  • NanoSonic leads the industry in anti-corrosion coatings.
  • PSI uses tunable diode laser absorption technology, among others, to develop low-cost, high-volume applications such as natural gas leak detection and greenhouse gas monitoring.
  • XIA supplies advanced digital spectrometers for x-ray, gamma-ray, and other radiation detector applications to research universities, National Laboratories, and industry.
  • Vista Clara develops nuclear magnetic resonance instrumentation that delivers quantitative imaging of subsurface hydrogeologic structure.
  • Adelphi Technologies produces a range of high-energy neutron sources for industrial and research applications.

These knowledge effects are also reflected in survey responses (see Box 7-3) and in the sometimes intensive use of patenting to protect SBIR-derived IP. For example, LI-COR is the assignee for 89 patents published between 1981 and 2015, according to the U.S. Patent and Trademark Office.

Innovative Technologies and Product Development

For the majority of recipients, the SBIR/STTR programs support work on their core technology. At least initially, few companies are large enough to advance multiple technologies simultaneously, although some companies are working on platform technologies that can be further developed into products that share a core technology. Over time, companies may grow to the point that they can support multiple projects but are still small enough to qualify for SBIR funding, which can then be used more selectively.

Many of the survey respondents and case study executives noted that the SBIR/STTR programs have provided critical support in developing core innovations and platform technologies (see Box 7-4). There is evidence that for smaller companies in the energy sector, funding for unproven core technologies is difficult to acquire outside of the SBIR-STTR programs. According to the PWC MoneyTree survey of venture investments, the number of seed-stage venture capital investments has declined steadily over the past decade, and in the second quarter of 2016, the industrial/energy section accounted for 3 percent of the total venture capital investment in all industries.5 For 2015, the Center for Venture Research reported that 11 percent of angel funding deals were in the industrial/energy sector.6 The Center

___________________

5 PWC Moneytree Survey, https://www.pwcmoneytree.com.

6 Jeffrey Sohl, “The Angel Investor Market in 2015: A Buyers’ Market,” Center for Venture Research, May 25, 2015.

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

also reported that the share of funding devoted to the seed stage declined sharply from 46 percent in 2013 to 25 percent in 2014.7 In 2015 there was a 1.9 percent increase in the amount of investments.8

___________________

7 Jeffrey Sohl, “The Angel Investor Market in 2014: A Market Correction in Deal Size,” Center for Venture Research, May 14, 2015.

8 Jeffrey Sohl, “The Angel Investor Market in 2015: A Buyers’ Market.”

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

Connecting to Research Institutions

Although the STTR program is specifically designed to connect small companies to research institutions (RIs), this objective is also accomplished to a considerable degree by the SBIR program. Survey data are provided in Chapter 5, but case study meetings and survey comments underscored the closeness of the connection for many companies. For example, XIA maintains research relationships with a broad range of academic, government, and corporate entities such as University of California, Davis, University of Texas at Austin, Michigan State University, Pacific Northwest National Laboratory, Los Alamos National Laboratory, Lawrence Livermore National Laboratory, Institute for Nuclear Physics (Germany), Radiation Protection Bureau, Health Canada, Alameda Applied Science Corporation, and IBM.

Multiple relationships exist for a number of SBIR/STTR firms, especially those that have grown over the years.

  • Creare has strong relationships with machine equipment companies such as KMT, MAG IAS, Fives, Harris Aerostructures, Saint-Gobain, Guhring, Iscar, AMETEK/Precitech, and many others.
  • NanoSonic reports good relationships with at least eight universities and effective partnerships with Colorado State University, the Naval Postgrad School, and the University of Arizona.
  • In addition to the National Laboratories, Muons has partnered with eight universities: Cornell University, University of Chicago, Florida State University, Hampton University, Illinois Institute of Technology, North Carolina State University, Northern Illinois University, and Old Dominion University.

None of the case study companies focused exclusively on STTR. Those that partnered with RIs using STTR also partnered with those RIs, but on different terms, using SBIR.

Leverage Effects

Many DoE companies develop tools and instrumentation that are used primarily or exclusively by researchers to great effect. The enabling, though indirect effects, are in many cases substantial and represent an important way that knowledge generated through SBIR is leveraged to gain wider currency in the scientific community.

SBIR/STTR companies that serve the research community provide products and services rarely at the scale needed for a direct large commercial success. Nonetheless, they can be of enormous importance to the innovation ecosystem as whole, providing resources that others use to address large-scale problems. A few examples include the following:

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
  • Dr. McDermitt (LI-COR) estimated that greater than 80 percent of the measurements of the carbon balance of agricultural and natural ecosystems have been made using LI-COR instruments, noting that “much of what we now know about how climate change might influence ecosystems comes from data provided by these instruments; it’s made all this scientific work possible.”
  • Calabazas Creek Research provides instruments to several of the National Laboratories.
  • XIA’s gas technology enables core functions for the massive Stanford Linear Accelerator Center (SLAC) accelerator (as well as others).
  • Creare and PSI have provided numerous bespoke tools and technologies to the research community.
  • Vista Clara’s hydrological tools enable detailed mapping of declining water resources.

COMPANY IMPACTS

For many small companies—especially those that receive SBIR or STTR funding early in their history—receiving an award (especially Phase II) can be a highly positive or transformative experience. One-third of the respondents to the 2014 Survey indicated that program funding transformed the company. Appendix E details how SBIR/STTR funding affected the trajectory of development for each of the 12 companies studied. This section describes some of these impacts.

In general terms, the 12 case study companies described how SBIR funding made an especially significant difference early in the company’s existence. For example, Dr. Warburton said that SBIR/STTR funds were critical to XIA’s founding and growth because funds were not available from other sources. Mr. Nemser said that the SBIR program was not only CMS’s first funding source, but also, in subsequent years, the only funding source that supported exploration of a range of possible applications for its technology.

Many of the 2014 Survey respondents stated quite bluntly that the company would not be in existence without the SBIR program (see Box 7-5). As expected, many of these respondents were from very small companies with limited access to alternative funding, at the time of award and at the time of the survey. However, several respondents were from companies that had moved far beyond the SBIR program.

Profiles of individual companies provide a more nuanced view and illustrate both the difficulties of raising very early funding and the critical role of the SBIR program in filling this gap. Many of the case study companies and a considerable number of survey respondents described major difficulties in raising funds before their products reached the market. For many companies, the road to a successful product is long and expensive. Private

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

investors are often reluctant to assume the risks involved, which can be substantial even for companies that raise significant outside funding. The SBIR/STTR programs were designed to fill some of this gap, by providing funding that can be used to validate and demonstrate products.

The SBIR/STTR programs also assist in company formation because seed funding is also difficult to find. Venture capital firms have exhibited an increasing preference for supporting more established companies and technologies. The case studies (and survey responses) underscored the many types of projects for which alternative funding sources are scarce.

Companies also pointed out that the SBIR program plays a special role in funding projects that do not align well with commercial imperatives facing some large markets and the need for relatively quick returns.

Validation Effects

Often, the SBIR/STTR programs provide a unique mix of validation and funding for the acquisition of preliminary data needed to persuade potential partners that the technology has value, the management team is competent, and the company is sufficiently stable to partner with it.

The DoE SBIR program provides sufficient funding for product development in only some circumstances. However, case studies and survey responses illuminated the ways in which SBIR/STTR awards can provide the

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

technological and commercial validation that underpins acquisition of funds from other sources (see Box 7-6). In particular, companies stressed that the SBIR program can provide the necessary confidence that peer review provides, while the provision of non-dilutive funding sets the stage for successful efforts to raise funds in the private sector.

It is apparent from both case study meetings and survey responses that, through the SBIR/STTR programs, DoE in effect performs important due diligence on behalf of subsequent investors and customers. The peer review provides a technical assessment that even a well-established venture capital firm would be hard pressed to match, making the investment less risky.

Niche Markets

Innovative companies, especially small innovative companies, are often driven at least initially by the passion of the founder or founders to make a difference. What they often find is a substantial gap between technical success and commercial success, and between meeting the needs of the technology users and creating a sustainable or successful business.

This is especially true when the market being served is small either in numbers or resources. Outside (and particularly venture capital)

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

investors are often less interested in investing when markets are small and lower revenues make it less likely that the product will eventually generate significant commercial returns.

Many of the case study companies address the needs of highly specialized markets. For example, Woodruff Scientific focuses on fusion, and Calabazas Creek and Muons focus on high-energy physics.

The products that these companies provide can have an enormous leverage effect. For example, some of XIA’s technologies are necessary for the effective operation of projects run at SLAC and other accelerators. Even though the market is small, the $500 million annual investment in these projects requires XIA technologies to run effectively.

Long Cycle/High Risk

The SBIR/STTR programs also help companies to develop products and capabilities that require a longer research cycle than can be sustained with private funding. Most venture investors expect to see products succeed or fail within a few years, but some technologies take much longer to reach the market. Mr. Nemser (CMS) observed that DoE topics support longer sequences of work that develop platform technologies that permit a range of applications. A survey respondent explained:

I have chosen to pursue challenging and long-range technologies and hence might appear to have limited commercial success (when compared to other small businesses engaged in more rapidly commercialized technologies such as medical devices, software etc.). But developments in High Energy Physics are also useful to society, but on a longer time scale.

Several survey respondents stated that little funding is available for long-cycle or high-risk research aside from SBIR. One respondent wrote:

The SBIR/STTR program is effectively the only significant early stage non-dilutive funding source for long incubation time, high risk, high payoff technologies, such as advanced materials and manufacturing technologies, that could not be developed or commercialized by private enterprise without the program.

Another respondent explained that SBIR complements rather than replaces venture funding:

If a technology can be developed in a few years to reach a significant market, then private investors will probably find a way to fund it. Thus, the SBIR program should probably have a spottier record than a VC firm, judged on return on investment.

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

The potential long time to reach market presents a challenge in cutting-edge research.

Several of the case study company founders have been working on their projects for more than 20 years, and some have not yet reached scale in the market. Long-cycle research is difficult because revenues are delayed as costs mount. The core funders of advanced research besides the government (strategic partners and venture capital investors) are increasingly reluctant to fund projects that are not well along the path to market.

The DoE SBIR/STTR programs therefore support the early development of technologies that may have substantial social or even commercial value downstream, but which are too far from the market to attract other funding sources (see Box 7-7).

Platform Technologies

SBIR/STTR funding also supports the development of platform technologies (that is, technologies that can provide the basis for multiple products in different markets). Venture investors typically prefer companies to focus on a single technology for a single market, because the pursuit of multiple objectives could lead to loss of focus and eventual failure. However, platform technologies can provide the basis for work in a range of applications—many of which can be successful. For example, since successful

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

development of its core SBIR/STTR-funded technology, Diversified Technology Inc. (DTI) has built applications for radar, high-energy physics, and food and wastewater processing. In 1998, DTI received SBIR funding from the Navy for an advanced radar system. In 1999, DTI received multiple Phase I and subsequent Phase II awards to assist the Stanford Linear Accelerator Center in adopting solid state switches and sources. In 2003, SBIR funding from the Environmental Protection Agency allowed DIT to investigate the application of Pulsed Electric Field (PEF) processing to waste water treatment.9

Hiring and Staffing

Retention of high-quality technical staff is a perennial challenge for small innovative businesses. Such businesses are challenged by the long period between the start of research and the deployment of a product in the market. Because the company is not earning revenue during this period, it must seek other funding sources, which places it at risk of losing key staff if funding gaps occur. A long gap can mean the elimination of key staff who may not be available if funding resumes.

The SBIR/STTR programs improve the certainty of funding over at least a 2-year period (for Phase II), which provides small businesses with the confidence and means to hire and retain staff (see Box 7-8.)

___________________

9 Floyd Arntz et al., “New Concepts for Pulsed Power Modulators: Implementing a High Voltage Solid-State Marx Modulator,” http://www.divtecs.com/data/File/papers/PDF/ILC_Long_Pulse_Marx.pdf; “Advanced Solid State High Repetition Rate Modulator,” 1998, https://www.sbir.gov/sbirsearch/detail/147844; “Wastewater Treatment by Pulsed Electric Field Processing,” 2003, https://www.sbir.gov/sbirsearch/detail/147910.

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

PROGRAM MANAGEMENT

In general, the feedback on program management is positive: participants report that the DoE program is run efficiently, is reasonably transparent, provides solid technical feedback to applicants, and offers a useful array of supports at different stages. In particularly, they note that contract and award management is good, and topic managers are available when needed. Many of the case study meetings were with scientists and executives who work with other agencies and therefore could provide a comparative perspective.

For example, Dr. McDermitt (LI-COR) found the DoE SBIR program to be managed effectively. He said that the proposal process was clear, the letter of intent process was not too burdensome, and, aside from the enforced no-contact during the application process, the project managers were readily available for discussion. The administration of grants and the necessary level of documentation were reasonable and workable. Overall, he considered the DoE SBIR program to be a good program with which to work.

Dr. Johnson (Muons) agreed that the DoE STTR/SBIR programs are well managed. Recent changes, such as the introduction of letters of intent to allow for timely reviewer selection and the well-designed timeline on the agency website, he said, were welcome improvements.

Two major concerns emerged from these discussions. First, topics are often not aligned with the enhanced commercial thrust of the program, which may be channeling applications too tightly into a framework that favors some types of commercialization over breakthrough technologies. Second, the program is disconnected from the rest of DoE and offers no pathway to possible commercial opportunities within or related to the agency. These concerns are discussed in more detail in the relevant sections below.

Applications

The 2011 congressional reauthorization has resulted in more reporting and a more complex application process for companies, observe many survey respondents and case study firms. The amount of effort required to submit a proposal has, according to Dr. Green, more or less doubled even for a highly experienced company such as PSI. In his opinion, this represents a major barrier to entry into the program. Dr. Green noted that the 200-page SBIR/STTR instructions on grants.gov may partially explain why the number of applications is declining. Every SBIR/STTR proposal requires PSI to upload 10-30 different sections. One survey respondent said, “Given this situation and the excessive time required to keep current with all these sub-agencies, our company is becoming less and less enthusiastic in participating in the SBIR/STTR program.”

Dr. Green also noted that dealing with the government has generally become more difficult. Numerous forms and statements related to fraud and

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

abuse are now required: companies must provide documentation for every piece of equipment it plans to buy and must prove that it is actually paying everyone that it planned to pay.

A survey respondent made a similar point:

The SBIR/STTR Phase I and Phase II proposal process has become increasingly cumbersome and difficult. To submit a Phase I or Phase II proposal, and subsequently do business with the federal government, a company must be registered with ASAP, grants.gov, FedConnect, FSRS, PAMs, SAM, SBA, etc., and further interact with the agency contracting/grants officers. Every year adds more levels of bureaucracy to participate in the SBIR/STTR program.

Another respondent said,

The application procedure is very complicated and replete with opportunity for mistakes and missed deadlines. The myriad of government websites that are part of the process are overbearing and difficult to understand...It is almost as if Grants.gov was designed to obfuscate and complicate the application process.

Finally, another respondent concluded,

The cost of preparing a proposal compared to the probability of receiving an award has become quite high.

These issues may also help to explain why the numbers of SBIR/STTR applications have been declining across all agencies.

Dr. Gary (Adelphi) noted that agencies that provide more than one funding deadline annually, such as the Department of Defense and the National Institutes of Health, were better attuned to the speed of technical development, and suggested that DoE should consider adding at least one additional funding deadline annually.

Topics

Topics were the single area of greatest concern for the companies interviewed for this report.

  • Noncommercial topics. Several companies said that some DoE topics are clearly designed to address specific technical needs of the topic manager and have little commercialization potential.
  • Unfunded topics. Some companies observed that DoE wastes company time and resources when it seeks applications for topics that are not funded and suggested that DoE should focus its energies on topics that will be funded.
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
  • Topic development. Some companies noted that the topic development process is quite opaque to outsiders.

Dr. Gary (Adelphi) observed that DoE SBIR/STTR topics are in some cases clearly derived from the science-oriented interests of topic managers, while others also reflect a commercial interest. Adelphi initially won a series of more science-oriented awards but as a result of increasing internal focus on commercialization has become more selective about the topics to which it applies. However, some of its recent awards for neutron optics were in topics that showed limited commercial potential given market realities for that technology. A survey respondent made a similar point:

SBIR topics in DoE and DoD have become extremely focused on narrow agency programs and yet demand substantial commercialization potential without the realization that their focused topics have no substantial commercial potential outside of the specific agency needs.

Dr. Gary also expressed concern that some topics are simply not funded. He believed that DoE should not publish topics with no record of funding. One survey respondent noted,

[T]he agency asks for a specific topic, we suggest a solution, we receive enthusiastic reviews from reviewers, but not funding. Somewhere in the process there is a disconnect between the author of the topic and funding sources.

In addition, Dr. Gary said that the topic development process at DoE is quite opaque, and he suspects that for a number of topics the process is largely driven by research scientists within DoE. Although this approach may result in interesting science, he believes that it does not align with commercial opportunities: that is, not all good science is commercially viable.

Dr. Ives (CCR) noted that the wording of some topics does not change from year to year and that some of these topics have not been funded, which suggests that the agency is not interested in them. Because the application process requires substantial company resources, DoE should eliminate topics that are systematically not funded, .

XIA is seeing fewer topics that are potentially viable under current SBIR evaluation procedures, according to Dr. Warburton. Although DoE scientists seek tools and instruments that will enhance their research capability, these generally have extremely limited commercial potential and hence fail DoE’s “return on investment” criteria. For example, one recent topic was clearly designed to develop an instrument for use within one of the four accelerators that exist worldwide, which has almost no commercial potential.

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

Dr. Warburton (XIA) said that DoE topic managers continue to regard SBIR/STTR as a tax on their research funding, and therefore seek to use it to provide tools or technologies that could further their own scientific interests and programs. These topic managers have no interest in commercial potential and therefore do not review topics for commercial potential before the solicitation is released.

Dr. McDermitt (LI-COR) said that DoE topics in environmental research have been closely attuned to cutting-edge environmental research, partly because the previous program director built close relationships with that community over many years. DoE has also been open to ideas for new topics.

Mr. Kempkes (DTI) was critical of the DoE topic development process. He said that many topics remain unchanged from year to year. In addition, because the topics provide insufficient information on which to develop a proposal, it is necessary to contact the technical staff interested in the topic to better understand what is really being sought. Although DoE publishes the contact information for subtopic managers, they are, according to Mr. Kempkes, usually not the technical staff (mostly at the National Laboratories) who drive the topic, which complicates the process.

DTI remains concerned about unfunded topics and about proposals that are marked fundable but are not funded. Mr. Kempkes observed that DoE SBIR/STTR is selective in the way that Harvard is selective—that is, there are invisible processes and perhaps a lottery occurring behind the scenes. DoE does not publicly prioritize its topics, and as a result DTI has written proposals to address a topic only to find out later that “DoE didn’t care about that anymore.” Because they compete with each other, not all subtopics are funded.

Dr. McDermitt (LI-COR) strongly supports the idea of providing an “open” category in the solicitation (currently available for most DoE divisions but not the Office of Energy Efficiency and Renewable Energy [EERE]). He had, for example, looked at the current solicitation and found nothing of relevance to LI-COR.

Letters of Intent

In general, interviewees were strongly positive about the letter of intent (LOI) process and believe that it generates useful feedback for applicants. They suggested that DoE find ways to make the letter more detailed and thus provide more guidance.

Dr. Ives (CCR) said that the LOI process provides a good opportunity for companies to explore possible applications without committing substantial resources.

Mr. Nemser (CMS) observed that, although he never likes rejection, he would rather receive rejection of a 1-page document than a 20-page proposal.

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

Commercialization Benchmarks and Expectations

The new commercialization benchmarks—which include those mandated by Congress and implemented through Small Business Administration (SBA) guidance, and those developed by DoE for use during the application process—have clearly had a substantial effect and were generally welcomed by the interviewees.

Dr. Ives (CCR) supports the new SBA commercialization benchmarks for awardees with a minimum number of awards as a way to encourage firms to take a more commercial approach to their activities.

Dr. Johnson explained that, from its founding in 2002 until 2010, Muons mainly focused on muon collider particle research and development of related technology. It used consulting contracts and SBIR/STTR awards to fund this work. Introduction of the new SBIR/STTR commercialization metrics after reauthorization nearly bankrupted Muons, according to Dr. Johnson. In 2011-2012, the company was designated as not commercial and hence SBIR/STTR funding dried up, leading to layoffs. In 2010, the company started to explore Accelerator Driven Subcritical Reactors (ADSRs), which have become a thrust of its commercialization efforts. The company ramped up its commercial activity, winning contracts from Fermilab to upgrade one of its flagship experiments and with Toshiba and Niowave to build magnetrons. As a result, Muons is once again considered by DoE to be a commercial company eligible for SBIR/STTR awards.

Dr. Johnson remains concerned that the DoE SBIR/STTR programs appear to expect commercial outcomes soon after the conclusion of a Phase II award. He noted that a typical time from conception to start of payback even in large commercial enterprises is close to 9 years. A survey respondent also made this point:

[T]he time for a new technical idea to be turned into a product with return on investment by large companies is about 9 years. It is unrealistic to think that a small company without experience in marketing or sales staff, etc., can do better. To saddle a very small company with additional commercialization tasks, when developing a new technology typically requires more effort than the grant covers, is unfair and likely to fail.

Dr. Green said that PSI was already evolving toward a more pronounced focus on commercialization before the SBIR/STTR programs made changes in the same direction. Today, PSI is a strong supporter of the program’s shift away from research-only projects. The company no longer looks for only projects that it can win. Before they write a Phase I proposal, staff must have a commercialization plan; it is part of the bid decision for PSI.

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

Review

Overall, companies reported a positive opinion of the review process. Many said that the quality of the technical review is generally good and that the technical feedback is helpful. For example, one company executive said, “DoE has an excellent reputation for running an open program where each proposal is evaluated on its merits, with high-quality technical review.”

Criticisms focused on the following:

  • The lack of transparency in funding decisions
  • Uneven quality of commercial review
  • Overly specific demands for commercialization plans
  • Anonymous reviewers and lack of supervision

Dr. McDermitt (LI-COR) was positive about the review process. His company has a good success rate and therefore no major complaints. LI-COR has not received a review in which the reviewer missed the point, which has happened with peer-reviewed papers. Some reviews offered significant insights to important questions, which improved the project.

Dr. Gary (Adelphi) stated that insufficient information is provided to applicants about funding decisions—in particular, too many applications are graded as excellent but not funded. He is a strong proponent of better feedback more generally. For example, NIH provides an online resource (ERA Commons) where applicants can find all of their applications and reviews. In contrast, DoE applicants must submit a request to have a review sent to them, the window for which is limited.

Dr. Walsh (Clara Vista) said that he believed that DoE reviews in some cases rely too heavily on academic reviewers. He found that proposals could be downgraded if they did not include an academic partner.

Dr. Warburton (XIA) said that the DoE process lacks a consensus-building mechanism for reviewers, which in part stems from the lack of in-person interaction among them. In contrast, face-to-face (or phone conference) meetings of NIH review panels boost the effectiveness of the review overall.10 In particular, the discussions between the reviewers quickly expose the strengths and weaknesses of the arguments of the applicants and reviewers.

Mr. Nemser explained that CMS strongly prefers application systems that generate quantitative scores. This allows companies to see their place along the funding line. DoE provides strong technical feedback, but its program would be improved by clearer scoring and clear information about paylines.

___________________

10 See Chapter 2 (Program Management) in National Academies of Sciences, Engineering, and Medicine, SBIR/STTR at the National Institutes of Health, Washington, DC: The National Academies Press, 2015.

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

More generally, one survey respondent said, “[P]roposals should be judged more based on innovation, company, PI, facilities, and equipment with less emphasis on connections and collaborations.” Another respondent criticized the use of university researchers: “[U]se of external (typically university reviewers), is a bad practice. These reviewers are not objective, and base their reviews based on their personal objectives and research.”

Some survey respondents stressed that the anonymity of reviewers has negative consequences: companies cannot tell whether they (or their technical approach) are systematically rejected by the same reviewer. One survey respondent said:

The reviewer not having his name revealed cannot be challenged and basically they can say what they want without oversight or checks and balances. Also, the reviewers are likely to be the same, year over year, and the company gets one or more reviewer that does not like it, the company is out of luck.

Commercialization Review

Some companies and survey respondents expressed concern that the commercialization plans demanded by DoE for Phase I and Phase II are not helpful.

One survey respondent said, “[A] commercialization plan to be part of the Phase I proposal seems completely unjustified given that the Phase I proposal in most cases is a feasibility study to just investigate the idea.” Another respondent said,

The emphasis on ‘making a business case’ has become excessive. Particularly with truly innovative concepts the risks are high and the future very fuzzy. In our experience, generating a business plan and pro forma documents at the level now required is more an exercise in creative writing than the production of a realistic road map for the future.

Dr. Warburton (XIA) sees a substantial disconnect between the science and technical needs of topic managers and commercialization review. XIA found it difficult to pass both reviews simultaneously. He also observed that explicit hurdle rates (projected internal rates of return for the project, which must be addressed in the application) present significant challenges to any company providing high-tech, low-volume scientific instruments.

Dr. Warburton also wondered whether DoE has ever compared actual commercial outcomes in funded Phase II projects to the outcomes projected in the submitted commercialization plans, to evaluate whether the present methodology has any predictive capability or is just an exercise in creative writing.

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

Mr. Kempkes (DTI) said that commercial review is generally poor and that the required commercialization plans are of little overall value. When developing a new technology, a company typically has some idea of its possible applications, but at the Phase I stage neither the company nor the reviewer has a clear vision of whether those applications will be useful. The market is simply too far away to address. By Phase II, a company should be thinking about transition, but sometimes even then it is too early.

Mr. Kempkes suggested that DoE focus its efforts on ensuring that companies develop a strong commercialization plan by the end of Phase II. Demanding significant work on a plan before then wastes the time of the applicant and the reviewer. He also noted that commercialization reviewers are of variable quality.

A number of survey respondents argued that the current emphasis on commercialization comes at the cost of potential important innovations (Box 7-9).

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

Rebuttal and a More Iterative Review Process

A number of case study company executives shared their frustration with the lack of a mechanism to rectify minor problems with applications upon first submission, which forces them into resubmission or rejection and hence into lengthy delays. However, Manny Oliver, the DoE SBIR/STTR Program Executive, explained that DoE can sometimes make deadlines flexible to allow companies to correct minor errors on their applications.

Several company executives suggested different ways in which the connection between applicant and study section review panel could be improved. Mr. Nemser (CMS) supports processes that would allow companies to address errors or omissions and respond to reviewer comments before reviews are finalized, perhaps through some version of face-to-face defense, to the extent possible. He also supports the resubmission approach adopted by NIH. Dr. Warburton (XIA) noted that DoE offers no appeal process and no possibility for formal resubmission (as at NIH).11 He is therefore a strong proponent of the idea that companies be provided the opportunity to respond in writing (one to two pages maximum) to reviewer comments before final decisions are made.

These opinions were echoed by comments from survey respondents. One respondent said, “For proposals that are recommended for funding and are not funded DOE should provide an improvement and resubmission opportunity.” Another respondent compared DoE’s process to that of NIH:

[C]ompared to the NIH's Review Panel approach, where the reviewers meet to discuss the proposals they have reviewed, DoE’s review system is often problematic since there is no process to correct reviewer errors or misunderstandings and the process is competitive enough so that even a single reviewer mistake can sink a proposal.

Along those same lines, another respondent explained,

Some comments from reviewers can be answered by a simple sentence, sometimes there is a misunderstanding, sometimes a point is just overlooked by a reviewer. It would be great if there was an option to respond to reviewers comments before the decision about award is made.

Finally, referring to other agencies’ resubmission processes, another respondent suggested that

___________________

11 Ibid.

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

the most important improvement that the SBIR program needs to make is to allow resubmission of a phase II after the initial evaluation . . . giving the technology a real chance and for submitters to address concerns of the reviewers.

Policy Innovations

Recent policy initiatives of the DoE SBIR/STTR programs are discussed more systematically in Chapter 3. This section provides feedback about innovations from companies.

Funding Mechanisms: Phase IIB

Dr. Walsh (Vista Clara) explained that the company received Phase IIB funding from DoE to develop a custom cable for down-hole data logging. It sought $300,000 from DoE and planned to invest $75,000 of its own capital, and, although not required by DoE, he believes that these matching funds helped the company win the award.

Mr. Nemser (Compact Membrane Systems [CMS]) said that changes to the DoE SBIR/STTR programs have been significant improvements. The program appears to be focusing away from science and farther downstream, and CMS strongly supports the introduction of Phase IIA and Phase IIB at DoE, because CMS believes that a single Phase II award is often insufficient to develop a marketable product. Phase IIB provides up to one-third of Phase II funding to support moving the product to market. CMS has also participated in the introductory accelerator program.

Other Innovations

Dr. Walsh (Vista Clara) strongly supported DoE’s approach to set aside part of the STTR budget to pay for articles in peer-reviewed publications, which is often a significant amount. DoE allows labor costs for preparing articles and presenting at conferences and publication fees for print journals, although these costs do have to be included in the initial proposal budget. He believes that other agencies should follow DoE’s lead in this area.

DoE also now allows companies to charge patent application expenses, to a certain limit, as direct costs. This is a very welcome initiative, according to Dr. Walsh, because such costs otherwise come directly out of the company’s profit.

Dr. Warburton said that XIA supports recent efforts to add broader topics, which are occasionally funded. XIA won a Phase I for a broader topic, although it did not go to Phase II.

Mr. Nemser said that the Phase II accelerator program helped CMS launch its oil dehydration systems. Their success underpinned further grants

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

focused on solvent dehydration, a technology which is being commercialized by capitalizing on the existing infrastructure and manufacturing capability.

Ms. Nemser explained that, under current guidelines, direct to Phase II12 excludes work completed under a previous Phase I—that is, the program only supports work completed by the company without SBIR/STTR Phase I funding. This seems to be an unnecessary barrier, because previous Phase I work may have entirely novel applications. He recommends that DoE be more flexible in this area.

Phase III Within DoE

DoE does not have a Phase III policy to support the commercialization of technology developed in the SBIR/STTR programs. Moreover, DoE has no plan in place to encourage (or track) the take-up of technologies sponsored by DoE within DoE itself.

One interviewee explained that a recent experience with a National Laboratory suggests that the Laboratories are not following the Phase III directives in the current SBIR law. More generally, he noted that Phase III is currently not seen as a responsibility of the SBIR/STTR program office, and it does not appear that it is the responsibility of any other office within the agencies.

Mr. Nemser (CMS) observed that DoE is very good at supporting companies through Phase I and Phase II, but not after: companies face the “valley of death” on their own. He said that venture investors understand the need to double down on their investments to get through this period before products are introduced into the market, but that DoE has no capacity in place to do so and has made no real effort to help in this area.

Dr. Walsh (Vista Clara) said that DoE’s interest in Vista Clara technology stems from its need to manage groundwater contamination more effectively. DoE facilities are currently spending hundreds of millions of dollars on soil and groundwater remediation, and Vista Clara technology offers significant upgrades on existing approaches. However, despite the funding and interest implied through SBIR awards sponsored by the Office of Subsurface Biology, Vista Clara has made no sales to DoE. Dr. Walsh observed that there seems to be no clear connection between the SBIR program and other parts of the agency. Thus, although there is a topic every year on subsurface characterization and remediation, there are no follow-on contracts for SBIR winners.

DoE should develop a better path for the commercialization of SBIR/STTR technologies, Mr. Kempkes suggested. Because acquisition is a

___________________

12 “Direct to Phase II” refers to a recent change that allows a company to bypass a Phase I award and apply directly for a Phase II award. Previously, it was required that a company first apply for and receive a Phase I award before they were eligible to apply for a Phase II award.

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

function of the National Laboratories, DoE should require all Laboratories to develop and publish a plan to integrate SBIR technologies into their programs.

These criticisms were also reflected in numerous comments from survey respondents (see Box 7-10).

Connecting to Topic Managers

Dr. Gary (Adelphi) said that connections with DoE staff tend to be very limited. Project liaisons appear to have other more pressing responsibilities, and in most cases there is almost no contact between DoE staff and the principal investigator (PI) or company representatives beyond the resolution of contracting issues. At best, DoE staff are of little help in finding potential markets for the technology within DoE, he said.

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

Mr. Kempkes (DTI) said that DoE SBIR topic managers are usually helpful—if the company contacts them. In his experience, one DoE staff member is often manager for 4-10 subtopics, and as a result they become quickly overwhelmed. The quality of email responses vary. To be fair, topic managers are spread too thin, and improvements in this area would be helpful.

Funding Levels

Given the recent increase in funding levels, there was less interest in this issue among both case study companies and survey respondents. Dr. Gary said that Adelphi would certainly consider applying for less funding if there was some benefit to doing so—for example, a higher likelihood of success. However, because this is currently not the case, the company designs the project to meet the funding available.

One survey respondent focused on projects that include use of expensive hardware. Noting that “programs with high technical content or high hardware content are severely limited by the SBIR funding limit,” the respondent recommended that projects involving substantial amounts of research hardware should be identified and funded at a higher level.

National Laboratories

Many of the case study companies have worked closely with the National Laboratories, and in several cases, founders had worked at National Laboratories for many years before starting their company.

Dr. Johnson said that most of Muons’ work focuses on identifying projects and technologies that will help the National Laboratories, but for which there is no available funding. Most other STTR projects work to transfer technology in the other direction, from the laboratory to the marketplace. STTR in particular has been used to meet those needs, perhaps acting as a DoE analog to Lockheed’s famed Skunk Works as a source of innovative technologies.

Dr. Warburton (XIA) said that SBIR-funded electronics to control spectrometers replaced the difficult-to-tune and expensive-to-maintain analog controls that had been the industry standard, leading to very substantial increases in efficiency for National Laboratories using the new technology. He noted that XIA has sold approximately $10 million to $20 million in instruments for synchrotrons, which cost $500 million to build and approximately $200 million annually to operate. A large percentage of the research undertaken with these systems requires instruments such as those developed by XIA. Synchrotron x-ray fluorescence experiments would not run at all without them, and overall productivity (and hence return on investment) would be a fraction of what it was today.

National Laboratories have few incentives to cooperate fully with small businesses, Dr. Warburton observed. In the best of cases, the Laboratory

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

scientists involved see STTR as a means of supporting their own research program, in exchange for providing the company with technical support. In other cases, Laboratory staff see the program as a means to generate funds and have no interest in commercial outcomes or even their partner’s interests.

Dr. Warburton noted that each National Laboratory has its own culture: XIA has worked quite successfully, for example, with Pacific Northwest National Laboratory and with a few departments at Lawrence Livermore National Laboratory, but essentially not at all with Lawrence Berkeley National Laboratory even though it is the closest of the three Laboratories to XIA headquarters.

The National Laboratories vary widely in their capacity to address SBIR, according to Mr. Kempkes (DTI). Some Laboratory staff have figured out how to use SBIR to advance their work. Others consider SBIR to be the competition. These differences are often more personal than institutional. It is not the case that some Laboratories are better to deal with than others; rather, it is the case that some contracting officers and some Laboratory staff are easier to deal with. Therefore, for example, SLAC contains the best partners DTI has found across the system, and also the worst.

One survey respondent indicated that contracting staff at National Laboratories are sometimes not familiar with SBIR: “National Labs [staff] should be trained to understand and implement the SBIR policy directive, and not attempt to appropriate intellectual property. My experience with legal personnel at Brookhaven was that they were completely ignorant of the goals and constraints of the SBIR program.”

Mr. Kempkes agreed that National Laboratory staff do a poor job of protecting company IP: the Laboratories are more academic than commercial, and hence they have a different mindset. Part of their job is to collect and disseminate information: “They live for publication.” Therefore, they are not mindful of keeping track of whose IP is whose. In every case, the company had experienced difficulties in creating an IP agreement with the National Laboratory.

Mr. Kempkes said that there are indirect challenges as well: SBIR proposals are considerably stronger when they are bolstered by letters of support from potential users. However, National Laboratories are not in a position to provide such letters because that would prevent a specific Laboratory from reviewing the proposal. The dual nature of the Laboratories as both customer and reviewer can therefore present a problem.

STTR

Several of the companies interviewed had extensive experience with STTR at DoE, and were able to provide more detailed insights especially into the relationship between STTR and the National Laboratories.

Dr. Johnson noted that STTR projects can only work well if there is goodwill between the Laboratory and the company. He noted that his

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

company, Muons, has long and deep connections with National Laboratories, its staff know most of their counterparts at the laboratories, so the connection is always positive.

Still, Laboratory administrators in general tend to view STTR awards as small projects. From a $150,000 award, the Laboratory will receive maybe $50,000 to $60,000, and it costs them almost that much just to do the paperwork, according to Dr. Johnson. So STTR agreements can take a long time to receive signoff from the Laboratories because they are a low priority for Laboratory administrations.

Dr. Gary (Adelphi) is a strong supporter of the STTR program and believes that companies are best positioned to determine whether a project should be SBIR or STTR, based on the needs of the project. He observed that a separate solicitation for STTR is likely to generate poor quality partnerships put together primarily to pursue funding, and that SBIR/STTR should provide a single opportunity for funding

Dr. Ives (CCR) sees STTR as an enormously helpful program and finds that, in some cases, it is a better vehicle for company initiatives than SBIR (in which the company also participates extensively). STTR provides an appropriate structure for partnering with RIs and offers access to the creativity and enthusiasm of graduate students. A recent STTR project with North Carolina State University led to the incorporation of student-developed designs into CCR products.

CCR has had differing experiences with RIs. Some research institutions, such as North Carolina State, offered realistic licensing terms and welcomed collaboration with small companies. Other RIs did not appear to understand the limited resources of small businesses and required unrealistic upfront licensing fees and royalties.

Dr. Ives said that DoE STTR grants for projects partnering with a National Laboratory used to require a cooperative research and development agreement (CRADA), but now they require only an IP agreement with the laboratory. An STTR grant also requires approval from the DoE Cognizant Officer who is responsible for Laboratory activities, which can take considerable time to receive. Currently, most Laboratories that use CRADAs require separate CRADAs for each of the two award phases, which lengthens delays and adds cost. Each CRADA specifies a time period for work to be completed, and amending this time requires a change to the CRADA, as does any other significant change to the statement of work (e.g., a shift to a different part of the Laboratory as provider of a device or service).

Partnering with RIs results in other challenges. In particular, universities and students want to publish their research. It was therefore, in Dr. Ives’ view, important to understand this need and provide opportunities to publish without compromising company IP. Dr. Ives believes this can be accomplished, as the record of publications related to CCR–university collaborations shows.

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

Dr. Johnson (Muons) observed that most companies do not want to deal with STTR grants—“We are masochists,” he said, “since most companies do not want to deal with National Lab bureaucracies and do not want to share their grant money with the labs.”

Dr. Green (PSI) is a strong supporter of the STTR concept. However, although STTR provides funding for the RI, industry has to be the bridge that transitions technology out of academia. STTR cannot just be pass-through funding to the RI. He believes that STTR encourages each partner to work to its strength: the RI does research and education, and the industry partner does commercialization, and this structure is perfect for technology transition.

PSI’s connections to RIs go far beyond STTR. Over a 6-year period, PSI funded 53 different universities. The company watches the scientific literature to identify possible partners, focusing on faculty who are making cutting-edge advances that can meet the needs of PSI’s customers. It is rare that a professor is not interested in collaboration.

Although Dr. Walsh (Vista Clara) does not object to partnering with RIs on occasion, he believes that, in most cases, Vista Clara could have done a better job without them. In only a few of the seven or eight partnerships formed for SBIR/STTR did the university add real value.

XIA has not had good experiences with the STTR program, Dr. Warburton explained. For example, a collaboration with Brookhaven National Laboratory worked out poorly, with no accountability for the project at the laboratory.

Mr. Kempkes said that DTI is wary of undertaking STTR projects, although it is currently in a partnership with Lincoln laboratory at MIT and has applied under some past STTR solicitations in partnership with Arizona State University (ASU). Typically, DTI does not bid on STTR solicitations unless there is something that the company cannot do itself (e.g., ASU grows algae). Usually, there is not enough funding in an SBIR to start with, and sharing the funding with a RI exacerbates this problem.

Dr. McDermitt explained that LI-COR’s experience with STTR has not been very positive. The company encountered a considerable amount of paperwork and issues related to IP that were difficult to resolve. As a result, the company has decided to not apply for STTR awards in the future. He noted that beyond STTR, LI-COR has continued to work with universities on a regular basis, and has developed a close relationship with the University of Nebraska–Lincoln (UNL). Li-COR Staff members worked as adjunct professors at UNL, the company has had numerous and valuable interactions with UNL, and two former staff members now work in the UNL Technology Transfer Office (TTO). LI-COR and UNL faculty worked together on a DNA sequencer in the 1990s, which was used on the human genome project and is still in use for protein detection and by LI-COR for the development of clinical applications.

In most cases, Creare directs STTR projects, Dr. Rozzi observed. However, a number of universities have established TTOs and incubators for

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

emergent small business concerns. Faculty are encouraged to form companies and work through the incubator. In these cases, they often seek companies such as Creare to partner on STTR proposals. However, Creare is very cautious about becoming involved in a partnership where the faculty member is the driver.

Dr. Lalli (NanoSonic) observed that, 5 years ago, she would have wanted to see the STTR program folded into the SBIR program, in large part because managing International Trafficking in Arms Regulations (ITAR) in the context of a partnership with an RI are often extremely challenging.

More recently, NanoSonic has found the process to move more smoothly. The clear tension that exists between academic interests in publishing and company needs for confidentiality can be addressed effectively with the right partner. Today, she said, NanoSonic is a very strong supporter of the STTR program. The company found a formal agreement to use university equipment to be very helpful. The program has helped NanoSonic to reach out to cutting-edge researchers and to gain access to high-quality graduate students.

ITAR

Dr. Rozzi noted that ITAR, the International Traffic in Arms Regulations,13 presents particular challenges to STTR. ITAR implements the Arms Export Control Act, which authorizes the President to “control the import and the export of defense articles and defense services . . . .”14 Creare has taken a very conservative view of ITAR and has found it difficult to ensure that universities understand and accept the relevant restrictions, particularly when there are considerable numbers of foreign students in most high-quality engineering departments.

Dr. Lalli (NanoSonic) also noted that the need to deal with ITAR is challenging. Most SBIR topics from DoD and NASA require compliance with ITAR, and NanoSonic is working to improve its capacity to deal with ITAR-related issues.

IP Conflicts

Dr. Rozzi (Creare) highlighted past conflicts over publishing results. RIs, academics, and graduate students all want to publish, which may lead to conflicts with the company’s need to preserve trade secrets. However, he noted that there are ways to publish without breaching disclosure limitations.

Dr. McDermitt (LI-COR) said that IP ownership is one of the most complex issues to manage in an RI collaboration. LI-COR wants to own the IP in part because it usually provides most of the funding. Some RIs are good to

___________________

13 22 CFR 120-130.

14 22 USC, Section 2778.

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

work with, and others are not: If the RI views STTR strategically as revenue generation opportunity, then significant conflicts and problems will almost always follow.

Partnerships with Research Institutions

STTR partnerships tend to be with RIs that are well known to Creare engineers, Dr. Rozzi observed. For example, Purdue, where Dr. Rozzi earned his PhD, is a top Creare partner. Creare has also worked closely with MIT in the past, but not as extensively in recent years. Similarly, another engineer had developed a close relationship with the University of Minnesota.

For Creare, the bar for involvement in STTR is simply higher than that for SBIR. Dr. Rozzi said that unless the RI is a great partner, paying the RI will not generate results that are nearly as efficient as if Creare did the work itself. STTR works best when Creare requires access to unique RI technologies—for example, previous STTR partnerships with Purdue provided access to modeling for composites machining. Because an RI cannot be easily made accountable or “fired,” Creare has to be very careful about entering into a partnership. Finally, STTR also requires an IP agreement, so if one is not in place and Creare does not have an existing relationship with the RI contracts staff, a considerable amount of work is needed before the proposal can even be advanced. Therefore, the partnership really has to be worth it, from Creare’s point of view.

OTHER COMMENTS

Because of the difficulties finding outside funding for further product development prior to revenue generation, Ms. Nemser said that the CMS’s top priority is for DoE to shift more funding downstream to help with commercialization.

Dr. Green (PSI) said that the agencies should simplify the application process and limit the amount of paperwork required. Every company should have a fair shot at an award, but this is not the case. Although it has a fully trained technical publications department, it still takes PSI significant time and effort to develop an application. It is important that the SBIR/STTR programs remain fully merit-based so that the best solutions find their way to the market.

Dr. Warburton (XIA) said that small instrument sales in support of the National Laboratories’ missions are in the national interest and that this class of SBIR topic should be assigned evaluation criteria that properly reflect their value to those missions. Alternatively, if DoE only wants proposals capable of large commercial returns, it should revamp its solicitations to bring them into conformance with that objective.

Mr. Kempkes (DTI) said that DoE prioritizes topics after the proposals are reviewed. He believed that this had nothing to do with proposal quality but simply reflects program need. He said that this process should be

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×

undertaken before publication of the solicitation, not after companies have expended hundreds of hours of effort on topics that are in fact low priority and unlikely to be funded.

Dr. Johnson (Muons) said that DoE program managers are quite flexible but are constrained by STTR legislation that requires that the RI receive a minimum percentage of the award. Program managers will sometimes allow replacement of an RI, which is usually not practical because the RI was selected for its specialized expertise. He suggested that program managers should be given the flexibility to revert STTR funding back to the company in special circumstances. One survey respondent recommended: “[T]he proportion of funds for the RI should be left to negotiations between the PI and the RI Program Manager. This could mean higher or lower amounts of funds going to each party.”

Mr. Kempkes (DTI) suggested that data rights should be extended beyond 5 years, which in terms of product development is far too short. For example, DTI worked on helicopter blades under an SBIR contract. It entered into negotiations with a helicopter manufacturer, which in the end decided to wait out the 5 years.

Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 151
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 152
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 153
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 154
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 155
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 156
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 157
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 158
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 159
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 160
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 161
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 162
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 163
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 164
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 165
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 166
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 167
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 168
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 169
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 170
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 171
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 172
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 173
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 174
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 175
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 176
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 177
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 178
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 179
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 180
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 181
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 182
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 183
Suggested Citation:"7 Insights from Case Studies and Survey Responses." National Academies of Sciences, Engineering, and Medicine. 2016. SBIR/STTR at the Department of Energy. Washington, DC: The National Academies Press. doi: 10.17226/23406.
×
Page 184
Next: 8 Findings and Recommendations »
SBIR/STTR at the Department of Energy Get This Book
×
 SBIR/STTR at the Department of Energy
Buy Paperback | $70.00 Buy Ebook | $54.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The Small Business Innovation Research (SBIR) program is one of the largest examples of U.S. public-private partnerships, and was established in 1982 to encourage small businesses to develop new processes and products and to provide quality research in support of the U.S. government’s many missions. The Small Business Technology Transfer (STTR) Program was created in 1992 by the Small Business Research and Development Enhancement Act to expand joint venture opportunities for small businesses and nonprofit research institutions by requiring small business recipients to collaborate formally with a research institution. The U.S. Congress tasked the National Research Council with undertaking a comprehensive study of how the SBIR and STTR programs have stimulated technological innovation and used small businesses to meet federal research and development needs, and with recommending further improvements to the programs. In the first round of this study, an ad hoc committee prepared a series of reports from 2004 to 2009 on the SBIR and STTR programs at the five agencies responsible for 96 percent of the programs’ operations -- including the Department of Energy (DoE). Building on the outcomes from the first round, this second round presents the committee’s second review of the DoE SBIR program’s operations.

Public-private partnerships like SBIR and STTR are particularly important since today's knowledge economy is driven in large part by the nation's capacity to innovate. One of the defining features of the U.S. economy is a high level of entrepreneurial activity. Entrepreneurs in the United States see opportunities and are willing and able to assume risk to bring new welfare-enhancing, wealth-generating technologies to the market. Yet, although discoveries in areas such as genomics, bioinformatics, and nanotechnology present new opportunities, converting these discoveries into innovations for the market involves substantial challenges. The American capacity for innovation can be strengthened by addressing the challenges faced by entrepreneurs.

READ FREE ONLINE

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!