An Assessment of the SBIR Program at the Department of Energy (2008)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

2 Findings and Recommendations I. NRC Study Findings The Department of Energy (DoE) SBIR program is making significant progress in achieving the congressional goals for the program. The SBIR pro- gram is sound in concept and effective in practice at DoE. With the programmatic changes recommended here, the SBIR program should be even more effective in achieving its legislative goals. A. Overall, the program has made significant progress in achieving its congressional objectives by: • Stimulating technological innovation; (see Finding G) • Using small business to meet federal research and development needs; (see Finding C and D) • Fostering and encourage participation by minority and disadvantaged persons in technological innovation; (see Finding E) and • Increasing private sector commercialization of innovations derived from federal research and development. (See Finding B.) These objectives are set out in the Small Business Innovation Development Act (PL 97-219). In reauthorizing the program in 1992, (PL 102-564) Congress expanded the purposes to “emphasize the program’s goal of increasing private sector commercialization developed through Federal research and development and to improve the Federal government’s dissemination of information concerning small business innovation, particularly with regard to woman-owned business concerns and by socially and economically disadvantaged small business concerns.” 25 

26 Sbir AT THE DEPARTMENT OF ENERGY B. The DoE SBIR program is focused on commercialization and has seen meaningful achievement. There are, nonetheless, opportunities for improvement in commercialization. 1. A significant percentage of DoE SBIR projects are commercialized to some degree. i. Reaching the market. NRC Phase II Survey data indicate that 41 per- cent of SBIR-funded projects reach the marketplace or have commer- cialization underway. Using a different methodology, DoE’s internal survey reported that 30.3 percent of firms reached the market. ii. Revenue skew. The NRC Phase II Survey data also show that a much smaller number (4 percent) of projects generate more than $5 million in revenues. The distribution of sales resulting from SBIR awards from DoE is not qualitatively different from the distribu- tion of returns from private sector investments. These few projects have, however, been extremely successful; in the case of Atlantia Offshore, for example, a single Phase I and Phase II pair of awards resulted directly in a product that generated over $500 million in sales, in addition to other societal benefits. DoE’s survey of their Phase II awardees indicated that 11 percent of companies reported sales greater than $850,000. iii. Licensing revenue. The NRC Phase II Survey indicates that licens- ing revenues have not been a significant source of additional com- mercial success at DoE. See Figure 4-1. Twenty-four percent of NRC Phase II Survey respondents reported products/ services/or processes in use at the time of the survey, and 18 percent reported commercialization underway (figures rounded). See NRC Phase II Survey, question 1. See NRC Phase II Survey, Question 4b. As with investments by angel investors or venture capitalists, SBIR awards result in highly con- centrated sales, with a few awards accounting for a very large share of the overall sales generated by the program.  These are appropriate referent groups, though not an appropriate group for direct com- parison, not least because SBIR awards often occur earlier in the technology development cycle than where venture funds normally invest.  Nonetheless, returns on venture funding tend to show the same high skew that characterizes commercial returns on the SBIR awards. See John H. Cochrane, “The Risk and Return of Venture Capital,” Journal of Financial Economics, 75(1):3-52, 2005. Drawing on the VentureOne database, Cochrane plots a histogram of net venture capital returns on investments that “shows an extraordinary skewness of returns. Most returns are modest, but there is a long right tail of extraordinary good returns. Fifteen percent of the firms that go public or are acquired give a return greater than 1,000 percent! It is also interesting how many modest returns there are. About 15 percent of returns are less than 0, and 35 percent are less than 100 percent. An IPO or acquisition is not a guarantee of a huge return. In fact, the modal or “most probable” outcome is about a 25 percent return.” See also Paul A. Gompers and Josh Lerner, “Risk and Reward in Private Equity Investments: The Challenge of Performance Assessment,” Journal of Private Equity, 1(Winter):5-12, 1977. Steven D. Carden and Olive Darragh, “A Halo for Angel Investors,” The McKinsey Quarterly, 1, 2004, also show a similar skew in the distribution of returns for venture capital portfolios. NRC Phase II Survey, Question 3. 

FINDINGS AND RECOMMENDATIONS 27 iv. DoE awards are characterized by a relatively high degree of pri- vate sales. Of the DoE SBIR award recipient firms that responded to the NRC Phase II Survey and reported sales of some type, 76 percent sold to domestic private sector firms and 14 percent to export markets. v. Limits on potential for broader commercialization. In cases where the market is inherently limited, as with the case with sen- sitive energy technologies, products developed with DoE SBIR assistance often cannot become a large commercial successes. 2. DoE has a substantial history of commercialization support and has expanded its efforts in recent years. i. DoE has sponsored a Commercialization Assistance Program (CAP) for the past 17 years. That program has provided, on a voluntary basis to Phase II awardees, individual assistance in developing busi- ness plans and in preparation of presentations to potential invest- ment sponsors. It is operated by a contractor, Dawnbreaker, Inc., a private firm based in Rochester, NY. ii. For the past 6 years, DoE has offered their Phase II awardees addi- tional market identification and networking service which requires much less time commitment from awardees than the very intensive CAP. These programs have provided professional assistance in business plan development and/or market evaluation to participating SBIR companies. iii. Systematic tracking is necessary to assess the impact of these efforts on commercialization. 3. Third-party investors can be encouraged by the validation effect of SBIR funding. i. Additional investments. Sixty-three percent of respondents indi- cated that they had received or made additional investments in the surveyed project, averaging just under $1 million per project.10 ii. No venture funding. For a number of reasons (at least until recently) the energy sector in general and therefore DoE SBIR projects have not been attractive to venture capitalists. No responding projects indicated that they had received venture capital funding.11 See Table 4-2. The case study of Diversified Technologies illustrates this phenomenon with regard to specialized transformers. See Appendix D. DoE SBIR publications and Web site; interviews with DoE SBIR staff. DoE calls for its contractors, Dawnbreaker and Foresight, to track their clients’ performance for at least two years. 10See NRC Phase II Survey, Questions 22 and 23. 11See Table 4-4. 

28 Sbir AT THE DEPARTMENT OF ENERGY TABLE 2-1  2005 DoE SBIR Initiatives in Support of Commercialization Technology Niche Virtual Deal Assessment Simulator™ CAP Trailblazer (TNA) (VDS) Contractor Dawnbreaker, Foresight Foresight Foresight Inc. Science and Science and Science and Technology, Technology, Technology, Inc. Inc. Inc. Start Date January 2005 January 2005 January 2005 January 2005 Completion Date December 2007 December 2007 December 2007 December 2007 Eligibility SBIR II only SBIR I only SBIR I or II SBIR I or II SOURCE: Department of Energy SBIR program publications and Web site. iii. Acquisition. In some cases, the technology developed had sufficient commercial potential that investors bought the grantee company outright. iv. Multiple other funding sources. Many grantees have found addi- tional funds from a wide range of sources, including angel fund- ing. Sixty-three percent of NRC survey respondents attracted some additional investment (excluding further SBIR awards). 4. Commercialization and selection. Commercialization potential is now formally recognized as a selection criterion at DoE; it accounts for one-sixth of the total score used in selecting applications for award. 12 C. The DoE SBIR program is operated in alignment with the depart- ment’s mission. It has the potential to contribute to the missions of the National Laboratories. 1. Effective mission alignment. All DoE awards appear to be selected primarily on the basis of their potential to advance knowledge and provide solutions in the field of energy. There is no evidence that DoE awards are made in fields outside those linked to the department’s mission.13 12Interviews with DoE SBIR staff. 13Possibly the most dramatic identified success from the program has been Atlantia’s SeaStar ® technology, a cost-effective alternative for the development of otherwise inaccessible small oil fields in deep water. The technology generated more than $500 million in revenues for Atlantia. 

FINDINGS AND RECOMMENDATIONS 29 2. Agency technical managers are deeply involved in topic develop- ment and selection. i. The impact of an SBIR project on DoE’s mission to advance economic and energy security is carefully considered during the s ­ election process. Awardees and DoE staff note that impact effects are an important component in every application. In all the cases examined, DoE SBIR funded projects related to the science and technology of energy. ii. Results from the NRC survey of Technical Topic Managers indi- cate that of the DoE projects surveyed, 70 percent reported that the project manager was involved in the generation of the topic that led to SBIR award. Fifty-eight percent of project managers reported involvement with the technology before Phase I began. 3. Limited participation of DoE National Laboratories in SBIR. i. While the National Laboratories are an important source of techni- cal reviewers for proposals (which is an important component of the administration of the SBIR program at DoE), the Laborato- ries themselves are not otherwise strongly involved in the SBIR program. ii. As a result, the potentially significant role of the National Labo- ratories as partners with SBIR-award-recipient firms is not fully realized. D. The SBIR program at DoE has provided significant support for small business, frequently acting as the impetus for project deployment and the foundation of new firms. 1. SBIR awards from the Department of Energy fund the develop- ment of technologies that, otherwise, might have developed more slowly, if at all. i. The project initiation decision. More than 80 percent of NRC Phase II Survey respondents reported that they would likely or certainly not have gone forward with their project in the absence of SBIR funding.14 ii. New firm formation. Twenty-three percent of DoE respondents indicated that their companies were founded entirely or partly because of an SBIR award. Of the NRC Phase I Survey respon- dents, 11 percent stated that their firms were founded or remained in businesses due to SBIR Phase I funding they were awarded. Growth within companies also occurred, with 34 percent of responding 14See Figure 7-1. 

30 Sbir AT THE DEPARTMENT OF ENERGY Phase I companies hiring one or more new employees as a direct result of an award.15 iii. Critical source of early funding. Case study work suggests that alternative very early stage venture funding for SBIR projects is nearly nonexistent. Without exception, all of the case study com- panies indicated that SBIR was vital to the development of their technology. Most suggested that the technology would not have been created if there had been no SBIR program. All credited SBIR as having played a significant role in the company’s formulation or development. 2. The DoE SBIR program supports the engagement of small busi- ness in federal R&D. i. Contracting mechanism. SBIR provides a contracting mechanism that keeps small firms engaged in the innovation system. Many small firms find other approaches to seeking federal or corporate R&D support to be either too costly from a contracting standpoint or require too great a loss of control to be viable. This appears to be one reason why SBIR continues to be enthusiastically supported by the small business community despite the fact that the award sizes have stayed constant for over a decade. ii. Application process as a filter. At the same time, the applica- tion process—though simpler than that for other federal R&D pro- grams—is nonetheless very demanding for a small firm, and the success rate of Phase I applications is typically at 18-20 percent. 16 3. The SBIR program also provides additional benefits for participat- ing small businesses. i. Networks and partnering. SBIR facilitates technological innova- tion through the creation of new partnerships and the strengthening of networks of innovators. Among case study firms, networking was identified as critical when attempting to find partners to assist with commercialization.17 ii. Alternative path development. Companies often use SBIR to fund alternative development strategies, exploring technological options in parallel with other activities. iii. Expand the company’s technology base. SBIR awards can help build a company’s technology base, allowing the company to pursue 15Case studies of NanoSonic and Thunderhead Engineering, and Diversified Technologies, Inc. 16DoE SBIR program data prepared for the Subcommittee on Technology and Innovation, Com­ mittee on Science and Technology, U.S. House of Representatives, June 26, 2007. This success rate is subject to some variance—as high as 25 percent and as low as 12.5 percent in some years. 17See Figure 7-2. 

FINDINGS AND RECOMMENDATIONS 31 an expanded suite of research projects, bring in researchers with new points of view and knowledge, and develop new markets.18 iv. Stimulate other forms of financing. SBIR awards can help a firm attract other forms of financing, such as funding opportunities from states (NexTech). Banks may also be more willing to provide busi- ness loans to companies that have received a Phase II grant, recog- nizing that a Phase II award is a good indicator of an innovation’s potential marketability.19 v. Enhance credibility with federal agencies. SBIR awards typi- cally lend credibility to a firm and enable it to earn other federal grants and contracts, including those obtained through Broad Area Announcements.20 vi. Enhance credibility with private-sector firms. SBIR awards can provide a “stamp of approval,” allowing smaller companies to ­better access large private sector companies, including government con- tractors, and attract private investors. An SBIR award can instill confidence that the smaller firm, and its product, have been vetted to some degree, thus raising interest. Once one large company becomes aware of an SBIR firm, its reputation grows and others are likely to hear about the firm, its product, and its area of expertise.21 E. Support for minority- and woman-owned firms. 1. No long-term trend is apparent. Data from SBA indicates that between 1992 and 2005, the Phase I share of woman- and minority- owned businesses at DoE averaged just over 20 percent, with a decline in the early years of this decade, followed by an increase, accelerat- ing in 2005 (see Figure 2-1). Data for Phase II are similar, although slightly lower. On average, woman- and minority-owned firms won 22.1 percent of Phase I awards from 2001-2005, and 19.1 percent of Phase II awards.22 2. Lagging application success rates. For Phase I, applications from woman-owned businesses have had a lower rate of success compared to all other groups—by approximately 3-10 percentage points—in 18See case studies of NanoScience, PPL, Diversified Technologies, Inc., and Thunderhead Engineer- ing in Appendix D. See also the case study of Pearson Knowledge Technologies in National Research Council, An Assessment of the Small Business Innovation Research Program at the Department of Defense, Washington, D.C.: The National Academies Press, 2007 Prepublication. 19See the Airak case study in Appendix D. 20See the IPIX and NexTech case studies in Appendix D. 21See the case studies of Atlantia, NanoScience, and NexTech in Appendix D. 22See Chapter 6. 

32 Sbir AT THE DEPARTMENT OF ENERGY 30 Percent of All DoE Phase I Awards 25 20 15 10 5 0 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 Year Woman-owned Minority-owned FIGURE 2-1  Shares of Phase I awards for woman- and minority-owned businesses, 1992-2005. SOURCE: U.S. Small Business Administration, Tech-Net Database. 2-1 every year except one. For minority-owned companies, the success rate is better than for woman-owned companies but still lags behind the “other” category (neither woman-owned nor minority-owned). For 2002-2003, the success rate of minority-owned businesses was consid- erably lower than that for woman-owned and all other businesses. 23 F. Understanding firms winning multiple awards. 1. Wide distribution of funding. i. SBA awards data indicates that between 1992 and 2005, 3,698 Phase I awards went to 1,459 different companies. The top 20 win- ners accumulated 556 Phase I awards, or 8 percent of all awards made during this period. No firm received 50 or more DoE Phase I awards. ii. Only five companies have been identified as receiving more than 15 Phase II awards between FY1992 and FY2003, and only two received more than 20, with the maximum being 21.24 23Ibid. 24The top 20 percent of winning companies together received 11.1 percent of awards. This is sig- nificantly lower than the Department of Defense. 

FINDINGS AND RECOMMENDATIONS 33 2. Some products, involving complex, multidisciplinary technologies, require multiple awards and a long gestation to develop. i. Products developed within by SBIR-awardee firms often require multiple SBIR awards and/or other sources of funding. Many firms work in complex technology areas, where capabilities in comple- mentary technologies are the basis for competitive success. ii. Developing prototypes of such products, with associated business plans and reliably estimated costs of production, can take a decade or more to complete. 3. Some SBIR firms with multiple awards achieve commercial impacts via spin-off firms. i. Some DoE-SBIR firms with multiple awards specialize in solving practical problems posed by government. The approach of Creare, Inc. in Hanover, New Hampshire, for example, is to create and transfer to spin-offs technologies that have particular commercial potential. As a consequence, Creare alone has spawned 15 spin-off firms employing over 1,500 people, with annual revenues reportedly in excess of $250 million.25 G. Knowledge creation and dissemination effects. 1. The SBIR program at DoE supports knowledge transfer in several ways. i. Patents. SBIR companies at DoE have generated numerous patents and publications, the traditional measures of knowledge transfer activity.26 Forty-three percent of the projects responding to the NRC survey reported at least one patent application. Of these, 37 percent reported receiving a patent related to the SBIR project. ii. Peer-reviewed publications. Forty-six percent of projects surveyed by NRC resulted in at least one peer-reviewed article.27 iii. Knowledge transfer from universities. The NRC survey also sug- gests that SBIR awards are supporting the transfer of knowledge, firm creation, and partnerships between universities and the private sector:28 25As a point of comparison, Xerox Technologies Ventures, the famed venture capital arm of Xerox Corporation, generated 35 spin-off firms over a comparable time period. 26See NRC Phase II Survey, Question 18. 27Without detailed identifying data on these patents and publications, it is not feasible to apply bibliometric and patent analysis techniques to assess the relative importance of these patents and publications. 28See Table 4-14 in National Research Council, An Assessment of the Small Business Innovation ­ esearch Program, Charles W. Wessner, ed., Washington, D.C.: The National Academies Press, 2008. R 

34 Sbir AT THE DEPARTMENT OF ENERGY o In more than two-thirds of responding companies at all agencies, at least one founder was previously an academic; o About one-third of company founders were most recently employed as academics before the creation of their company; o At DoE, about 34 percent of projects had some alignment with a university, through the use of university faculty as contractors on the project, use of universities as subcontractors, or employment of graduate students. iv. Indirect paths. There is anecdotal evidence concerning benefi- cial “indirect path” effects—that projects provide investigators and research staff with knowledge that may later become relevant in a different context—often in another project or even another com- pany. While these effects are not directly measurable, discussion during interviews and case studies suggest they exist.29 2. SBIR research quality. i. The NRC survey of technical topic managers (TTM’s) indicated that technical managers see research funded by SBIR as being largely similar in quality to the research funded under other programs. ii. Another survey question asked whether more high quality research proposals were received than could be funded. Just under two- thirds (62 percent) of DoE project managers reported more fund- able ­ projects than were funded.30 Success rates for SBIR Phase I competitions are consistently below 20 percent. In most years, the success rates have been about 15 percent. H. The DoE SBIR program has not benefited from regular evaluation. 1. Prior to the congressional legislation authorizing this study, no systematic, external program assessment had been undertaken at DoE. As a result, program management is not sufficiently evidence-based. i. Partly as a result of insufficient resources, the program has insuf- ficient data collection, limited reporting requirements, and ­limited analytic functions. This limits the program’s capacity for self- assessment and adjustment. 29For a discussion of the “indirect path” phenomenon with regard to the results of innovation awards, see Rosalie Ruegg, “Taking a Step Back: An Early Results Overview of Fifty ATP Awards,” in National Research Council, The Advanced Technology Program: Assessing Outcomes, Charles W. Wessner, ed., Washington, D.C.: National Academy Press, 2001. 30Thirty-one percent reported about the right number of proposals, and eight percent reported fewer fundable proposals than funds available. 

FINDINGS AND RECOMMENDATIONS 35 ii. This lack of assessment means that program management does not have adequate ongoing information about how their actions affect outcomes such as commercialization, knowledge generation, and networking. 2. Developing a culture of assessment. DoE has recently developed a system for tracking outcomes internally. This represents a positive step towards an assessment culture, but a range of issues still need to be addressed and a more systematic approach to evaluation adopted. I. Limited resources for program management. 1. The fraction of resources devoted to SBIR program management at DoE is the lowest among agencies studied by the NRC. i. Given the size of the program, the number of applicants each year, and the requirements of the award process, the SBIR staff at DoE is the minimum required to operate the program. ii. DoE SBIR staff devote nearly all their time to managing the pro- cesses for generating technical topics, and for receiving, evaluat- ing, and selecting grant applications. This leaves little time for other important tasks such as outreach, measuring Phase III activity, encouraging Phase III activity (both within and outside the depart- ment, including the national laboratories), documenting successes, understanding failure, and developing and implementing program improvements. 2. With most staff time devoted to compliance, there is limited oppor- tunity for program enhancement. i. Impact on assessment. Program managers do not have time or sufficient resources to support interaction between SBIR firms and technical staff; conduct internal and external assessment of the performance of funded firms; and evaluate the commercialization program and the overall effectiveness of awards. o Limited post-award follow up. While DoE’s technical staff have taken the lead in developing topics for the program solicita- tion, they have had very little interaction with the companies that receive SBIR awards.31 o Limited monitoring and assessment. Administrative staff or technical staff report few opportunities to visit funded firms or otherwise track their progress. Over the course of this NRC 31Interviews with DoE SBIR staff. Progress is being made in this regard. DoE technical staff has re- cently demonstrated its direct involvement in a newly instituted Continuation Application Process. 

36 Sbir AT THE DEPARTMENT OF ENERGY study, the DoE SBIR staff has sought new approaches to encour- age DoE’s technical staff to become familiar with the work of SBIR funded companies. o Limited evaluation. While the DoE SBIR staff maintain a detailed database of commercialization outcomes reported by funded firms, they do not as yet have the resources to validate the reported data or employ them in formal program evalu- ation. Evaluation of the effectiveness of commercialization programs has also been lacking. Program staff do not have adequate resources to assess the relative effectiveness of differ- ent approaches undertaken by DoE to enhance commercializa- tion outcomes. ii. Impact on cycle time. DoE provides only one solicitation annually. This contrasts in particular with DoD and NIH, which offer candi- dates several opportunities to propose projects. However, multiple solicitations require more resources, which are currently not avail- able to the at DoE SBIR program. iii. Impact on gaps between SBIR Phase I and Phase II funding. The funding gap between Phase I and Phase II also affects compa- nies. More than half the respondents to the NRC Phase II Survey reported that they stopped work on their project during this period. A small number of respondents (3 percent) reported ceasing opera- tions entirely in this unfunded interval. DoE has not adopted any of the measures to reduce the Phase I-Phase II gap implemented at other agencies.32 iv. Impact on outreach. Citing time and resource constraints, DoE SBIR staff decline most invitations to speak about the program. The agency does participate in the SBIR National Conferences, spon- sored by DoD and NSF. However, DoE has attempted to avoid state and local conferences, largely because the limited resources and available staff are focused on day-to-day program operations.33 II. NRC Study Recommendations The recommendations in this section are designed to improve the operation of the SBIR program at the Department of Energy. They complement the core findings that the program is addressing its legislative goals—that significant commercialization is occurring, that the awards are making valuable additions to nation’s stock of scientific and technical knowledge, and that SBIR is developing products that apply this knowledge to the Department of Energy’s missions. 32See NRC Phase II Survey, Question 28. 33Interviews with DoE SBIR program managers. 

FINDINGS AND RECOMMENDATIONS 37 A. Develop linkages between DoE technical staff and SBIR awardees. 1. Provide contact information. Communication with applicants, par- ticularly first-time applicants, is critical to the processes. DoE should increase the opportunity for all potential applicants to interact with the technical program managers, including the publication of their names and contact information within the annual solicitation.34 2. Provide pre-release information on topics. Pre-proposal communica- tion through a fair, transparent mechanism is likely to improve ­proposal quality and, therefore, overall program effectiveness. Consider the DoD Pre-Release information exchange model, whereby the relevant technical officer for each topic is available via email and phone for questions during a period before the official release of the solicita- tion. At DoD this practice has had the effect of ensuring not only that applicants have a better understanding of the work being solicited, but also in generating higher quality and better focused proposals. B. Improve commercialization support. 1. Develop a match-making function for SBIR awardees. Bring SBIR participants together with potential corporate customers, perhaps in trade show, technical challenge workshop, or technology demonstra- tion/validation formats. These functions could include large corpora- tions identified by the agency’s two SBIR commercialization assistance vendors, including large energy technology corporations that serve as DoE contractors. 2. A possible model. The Navy Opportunity Forum, initially based on a DoE initiative, is well funded. Similar levels of funding and scale might enhance DoE’s return on this type of activity. Another model is the National Renewable Energy Laboratory’s Enterprise Growth Forum. C. Explore reallocation of funding and topics between programs. 1. While SBIR funds are awarded by DoE using a competitive review of the best submitted proposals, the success rates of Phase I awards vary dramatically between the different programs. This disparity occurs because DoE allocates SBIR awards back to individual programs based upon a running average of the amount of SBIR funds contributed by that program, rather than only on proposal quality. 2.  DoE might consider whether current adjustments on an ad hoc basis are sufficient, or whether a policy that allows individual DoE programs to benefit from SBIR projects in excess of their contributed amounts would prove more effective. 34DoE initiated an information release process in FY2006. The relevant technical officer is available via email from the opening date to the closing date of the Funding Opportunity Announcement. 

38 Sbir AT THE DEPARTMENT OF ENERGY 3. By allowing DoE programs to compete among themselves for a greater share of total, SBIR can better address the agency’s mission and encourage proposals of the highest quality. D. Engage the National Laboratories: More outreach to the National Laboratories is desirable. This could include: 1. Subcontracting. Encourage participation of National Laboratories as subcontractors to the small business on SBIR projects, not least by removing regulatory barriers to the use of National Laboratories as subcontractors. More resources are likely to be required to carry out these activities. 2. Improve tracking. Develop procedures to track the relationship between National Laboratories and the SBIR program more formally, including the documentation of Phase III successes. E. Increase the participation and success rates of woman- and minority- owned firms in the SBIR program. 1. Encourage participation. Develop targeted outreach to improve the participation rates of woman- and minority-owned firms, and strategies to improve their success rates. These outreach efforts and other strate- gies should be based on an analysis of past proposals and feedback from the affected groups.35 2. Improve data collection and analysis. i. There appears to be room for further improvement in the participa- tion of women and minorities in DoE SBIR program. DoE should undertake efforts to assess why woman- and minority-owned compa- nies have experienced relatively lower success rates and to examine courses of action that may rectify this underrepresentation. ii. The Committee also strongly encourages DoE to gather the data that would track woman and minority firms as well as principal investigators (PIs), and to ensure that SBIR is an effective road to opportunity. The success rates of woman and minority Principle Investigators and Co-Investigators can also provide a measure of woman and minority participation in the SBIR program. 3. Encourage emerging talent. The number of women and, to a lesser extent, minorities graduating with advanced scientific and engineer- ing degrees has been increasing over the past decade. This means that many of the woman and minority scientists and engineers with the advanced degrees usually necessary to compete effectively in the SBIR 35This recommendation should not be interpreted as lowering the bar for the acceptance of ­proposals from woman- and minority-owned companies, but rather as assisting them to become able to meet published criteria for grants at rates similar to other companies on the basis of merit, and to ensure that there are no negative evaluation factors in the review process that are biased against these groups. 

FINDINGS AND RECOMMENDATIONS 39 program are relatively young and may not yet have arrived at the point in their careers where they own their own companies. However, they may well be ready to serve as principal investigators (PIs) and/or senior co-investigators (Co-Is) on SBIR projects. Over time, this talent pool could become a promising source of SBIR participants.36 F. Develop data for evaluation and conduct regular assessments. 1. Summary annual report. DoE should annually provide Congress with an enhanced summary report on the SBIR program. This should include descriptive statistics for applications, awards, and outcomes along the dimensions identified in this report, including knowledge creation, technology innovation, and impact on agency mission, as well as commercialization. As part of this process, DoE should produce regular reports from the commercialization database. 2. Company Commercialization Report. DoE may also wish to con- sider whether the DoD model—and technology—for producing a Company Commercialization Report, updated each time the company applies for further awards, might be a useful way of generating better data about commercial outcomes. 3. Regular assessments. The proposed annual report noted above could become a focus for wider efforts to develop improved internal assess- ment capabilities that can be used to enhance program operations. It could also tie proposed improvements to data-driven analysis and, for example, include an evaluation of the predictive power of selection scoring with regard to commercialization and other outcomes. DoE should also commission regular external arms-length evaluations to assess the program progress and the impact of new initiative. G. Provide additional management funding to develop and maintain a results-oriented program with a focused evaluation culture. 1. Effective oversight relies on appropriate funding. A data-driven program requires high quality data and systematic assessment. As noted above, sufficient resources are not currently available for these functions. 2. Increased funding is needed to provide effective oversight, including site visits, program review, systematic third-party assessments, and other necessary management activities. 36Academics represent an important future pool of applicants, firm founders, principal investigators, and consultants. Recent research shows that owing to the low number of women in senior research positions in many leading academic science departments, few women have the chance to lead a spin- out. “Underrepresentation of female academic staff in science research is the dominant (but not the only) factor to explain low entrepreneurial rates amongst female scientists.” See Peter Rosa and Alison Dawson, “Gender and the commercialization of university science: academic founders of spinout companies,” Entrepreneurship & Regional Development, 18(4):341-366, 2006. 

40 Sbir AT THE DEPARTMENT OF ENERGY 3. To enhance program utilization, management, and evaluation, addi- tional funds should be provided. There are three ways that this might be achieved: i. Additional funds might be allocated internally, within the existing budgets of the services and agencies, as the Navy has done, with commensurately positive results. ii. Funds might be drawn from the existing set-aside for the program to carry out these activities. iii. The set-aside for the program, currently at 2.5 percent of external research budgets, might be increased slightly, with the goal of pro- viding additional resources to maximize the program’s return to the nation.37 4. These recommended improvements should enable the DoE SBIR man- agers to address the four mandated congressional objectives in a more efficient and effective manner. H. DoE should consider the creation of an independent advisory board that draws together senior agency management, SBIR managers, and other stakeholders as well as outside experts to review current operations and achievements and recommend changes to the SBIR program. 1. Augmenting the role of the current DoE SBIR Oversight Committee, the purpose of such an advisory board is to provide a regular monitor- ing and feedback mechanism that would address the need for upper management attention, and encourage internal evaluation and regular assessment of progress towards definable metrics. 2. The annual report of the DoE SBIR program, recommended above, could be presented to the board. The board would review the report that would include updates on program progress, management practices, and make recommendations to senior agency officials. 37Each of these options has its advantages and disadvantages. For the most part, the departments, institutes, and agencies responsible for the SBIR program have not proved willing or able to make additional management funds available. Without direction from Congress, they are unlikely to do so. With regard to drawing funds from the program for evaluation and management, current legislation does not permit this and would have to be modified, therefore the Congress has clearly intended program funds to be for awards only. The third option, involving a modest increase to the program, would also require legislative action and would perhaps be more easily achievable in the event of an overall increase in the program. In any case, the Committee envisages an increase of the “set aside” of perhaps 0.03 percent to 0.05 percent on the order of $35 million to $40 million per year or, roughly, double what the Navy currently makes available to manage and augment its program. In the latter case (0.05 percent), this would bring the program “set aside” to 2.55 percent, providing modest resources to assess and manage a program that is approaching an annual spend of some $2 billion. Whatever modality adopted by the Congress, without additional resources the Committee’s call for improved management, data collection, experimentation, and evaluation may prove moot. 

FINDINGS AND RECOMMENDATIONS 41 3. The board could be assembled on the model of the Defense Science Board (DSB) or perhaps the National Science Foundation’s Advisory Board.38 In any case, it should include senior agency staff and the Director’s Office on an ex officio basis, and bring together, inter alia, representatives from industry (including award recipients), academics, and other experts in program management. 38The intent here is to use the DSB or the NSF Board as a model, not something necessarily to be copied exactly.