An Assessment of the SBIR Program at the Department of Energy (2008)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

5 Agency Mission 5.1 Managing a Program with Multiple Objectives Public Law 97-219 established SBIR with multiple objectives. The program is intended “to use small business to meet Federal research and development needs.” This is usually taken to mean that the SBIR program should be aligned with the R&D needs of the sponsoring agency. This objective, along with the other objectives mandated by Congress, must be translated into operational procedures for implementation. The existence of multiple objectives for the SBIR program means that agen- cies are forced to confront the following question: In selecting technical topics, and in selecting proposals for award, how much emphasis should be placed on satisfying the mission needs of the agency versus commercialization or one of the other SBIR legislative purposes? DoE’s overarching mission is “to advance the national, economic, and energy security of the United States; to promote scientific and technological innova- tion in support of that mission; and to ensure the environmental cleanup of the national nuclear weapons complex.” The Department has four strategic goals toward achieving this mission: • Defense Strategic Goal: To protect our national security by applying advanced science and nuclear technology to the nation’s defense. Department of Energy main Web site under “about us.” Accessed at <http://www.energy.gov/> on May 25, 2006. 72 

AGENCY MISSION 73 • Energy Strategic Goal: To protect our national and economic security by promoting a diverse supply and delivery of reliable, affordable, and environ- mentally sound energy. • Science Strategic Goal: To protect our national and economic security by providing world-class scientific research capacity and advancing scientific knowledge. • Environment Strategic Goal: To protect the environment by providing a responsible resolution to the environmental legacy of the Cold War and by provid- ing for the permanent disposal of the nation’s high-level radioactive waste. The SBIR program is structured to support all four mission objectives of DoE in a manner that parallels the overall allocation of R&D funding to non-weapons programs. Along with the other federal agencies that participate in SBIR, DoE attempts to strike a balance between these two purposes. However, at DoE, the link to agency mission—including the goal of performing quality science—is the pro- gram objective most clearly built into the program structure. During the Octo- ber 24, 2002, conference that helped launch the NRC SBIR assessment, Milton Johnson, the Deputy Director for Operations in the Office of Science, stated that DoE Office of Science “lives and dies by the quality of the science” produced. “If we produce lousy science, soon we won’t get much money for it.” In a May 2003 meeting, Dr. Johnson stated that SBIR was regarded within DoE like any other R&D program—that is, as a vehicle by which research programs could accomplish R&D objectives. According to Johnson, the difference with SBIR was simply that the R&D work was performed by small businesses instead of national laboratories or universities. However, Dr. Johnson also noted that the ability of small business to achieve excellent science is not the only measure of success. As a dual purpose program, the SBIR seeks not only to increase the involvement of small business in fed- eral R&D but also to increase private sector commercialization of innovations derived from federal R&D. He concluded that the view at DoE (based on the agency’s interpretation of the enabling legislation) is that commercialization will follow because it is in the best interest of the performers—the small businesses themselves. Throughout the history of the SBIR Program at DoE, the SBIR Office has promulgated a set of evaluation criteria that reflects this upper management Milton Johnson, “SBIR at the Department of Energy: Achievements, Opportunities, and Chal- lenges,” in National Research Council, SBIR: Program Diversity and Assessment Challenges, Charles W. Wessner, ed., Washington, D.C.: The National Academies Press, 2004. National Research Council symposium, “SBIR Program: Identifying Best Practice,” May 28, 2005, Washington, D.C. Interview with Arlene De Blanc, SBIR Program Analyst, Department of Energy, November 3, 2003. 

74 Sbir AT THE DEPARTMENT OF ENERGY philosophy, as articulated by Dr. Johnson above. That is, the likelihood of com- mercialization is included among the criteria for evaluating grant applications, but is outweighed by other criteria, which emphasize the scientific and technical merit of the proposal. 5.2 Alignment Issues for SBIR and the DoE Mission Despite the above approach to balancing the program’s dual purposes, the program has not always enjoyed unanimous support throughout all levels of management at the agency. The conceptual tension that exists between the two primary goals of the SBIR—to involve small firms in agency R&D and to fund projects with commercial potential—has resulted in a program that has earned increasing respect from program managers within DoE, and yet continues to receive relatively low levels of intraagency resources for administration. 5.2.1 Research vs. Commercial Culture Within the rigorous research culture that predominates within the Office of Science, managers of core research programs often are not in the habit of work- ing with small businesses, nor of considering commercialization to be a priority in the evaluation of grant applications. 5.2.2 SBIR as a Tax Furthermore (as in other agencies), those responsible for DoE R&D pro- grams have only slowly come to view the SBIR program as something other than a “tax” that draws resources away from more valuable activities. 5.2.3 Administrative Burdens Additional resistance also has derived from the relatively high cost and demanding nature of the SBIR program. SBIR proposals are less efficient to review, per dollar of research funded, because of the large number of proposals received, and more burdensome because of the tight deadlines. Also, the performers of the research are different from DoE norm—requiring more outreach than for grantees at universities, for example, where the process of grant applications is better understood. Finally, the agency’s peer-review system is labor-intensive; DoE conveys information packages to at least three reviewers for every proposal, selected for their expertise in the proposal’s subject matter, and retrieves them on time, or Particularly in the earlier stages of SBIR at DoE, this respect came grudgingly as some DoE pro- grams felt SBIR may not fit well into programs that have broad research missions. 

AGENCY MISSION 75 finds substitute reviewers. The peer-review process is now automated, however, and has presumedly resulted in greater efficiency in processing. 5.3 Changing Perceptions of SBIR There are indications that negative perceptions of the SBIR program among DoE technical staff are diminishing. Longtime SBIR staff member and former Acting SBIR Program Manager Arlene DeBlanc observed in an interview with the research team: “We make progress one [technical] program manager at a time. Once the [technical] program manager sees his needs are met, then he sees he can get value out of the program.” In this context, DeBlanc viewed an ultimate test of the program to be the willingness of at least some program managers to volunteer funds to SBIR in additional to the mandated set-aside. 5.3.1 Supporting Program Missions Interviews with several Technical Program Managers and Technical Topic Managers, who are not part of the SBIR program office staff,  confirm that sup- port for the SBIR program has grown within DoE: There was clear consensus among these managers that the SBIR research funded by their programs has sup- ported program missions, provided useful outcomes, and strengthened the role of small firms in those missions. 5.3.2 Providing Research Quality DoE has conducted two internal evaluations of the research quality of SBIR projects—one formal and one informal. The formal evaluation took place shortly after the department launched its SBIR program, at a time when considerable internal resistance to the program existed. Conducted by an independent program analysis office in the Office of Science (then the Office of Energy Research), that review was intended to compare the quality of SBIR research to other DoE- funded research, based on the assessments of DoE technical program managers. A former SBIR program manager recalls that the SBIR research was deemed by this study to be of slightly lower quality than other DoE funded research. These results were vigorously disputed by the SBIR office, which submitted a lengthy rebuttal. The informal study was conducted by SBIR Program Manager Robert Berger in the mid-to-late 90s. Dr. Berger attempted to survey DoE technical program managers to assess quality. In this survey, the SBIR research results were com- Interview with Arlene De Blanc, SBIR Program Analyst, Department of Energy, November 3, 2003. Interviews conducted by the NRC in March 2005. 

76 Sbir AT THE DEPARTMENT OF ENERGY parable to the non-SBIR research. This finding is consistent with anecdotal evi- dence based on interviews conducted as part of the present NRC study in March 2005 with TTMs and TPMs across several programs. The NRC project manager survey specifically attempted to assess the per- ceived quality of SBIR research by asking the project managers to rate the ­quality of research on scale where ten is best, one is worst for each of the SBIR Phase II projects in the survey. As a baseline, the project managers also were asked to use the same rating scale to assess the average quality of research of other-than-SBIR projects conducted for the project manager’s unit/office in the past two years. For DoE, the median rating was seven for both groups; the average rating was 6.80 for the SBIR and 7.24 for the non-SBIR group. (The standard deviation was 1.7 for both groups.) Further analysis showed that the result favoring non- SBIR research quality over SBIR quality is driven by outliers among the project managers who gave specific SBIR projects extremely low scores. As another measure of research quality, the project managers were asked whether their office received more high quality research proposals than they could fund. Sixty-two percent of DoE project managers reported that there were more fundable projects than they could fund, 31 percent reported that they received about the right number of proposals, and only 8 percent reported receiving fewer fundable proposals than they could fund. The results for DoD/NASA were similar. 5.3.3 Research Impact The project manager survey further asked whether the research conducted for the SBIR project made any difference in the way the project manager’s office conducted research or pursued other research projects. The limited impact here may be related to the fact that, unlike DoD and NASA, DoE does not procure technology for its own use; therefore, the impact of the research may be harder to identify. Of the 42 project managers who reported that SBIR-funded research had affected the manner in which their office had conducted research or pursued other research projects, about half encouraged the project performer to seek additional SBIR awards; more than half tried to follow up on project ideas in other research conducted or sponsored by the project manager’s office; some of these further efforts led to a blind alley. These internal findings are supported by an external review as well. In January 1989, the GAO reported that the quality of DoE’s SBIR research was comparable to non-SBIR research. U.S. General Accounting Accounting Office, Federal Research: Assessment of Small Business Innovation Research Programs, GAO/RCED-89-39, Washington, D.C.: U.S. Government Printing Office. Note: the total projects in the three categories exceed 42 because the project managers were in- structed to assign as many categories. 

AGENCY MISSION 77 5.3.4 Comparative Research Value Finally, in the survey project managers were asked to compare the value of SBIR research, per dollar spent, to that of non-SBIR research. DoE project m ­ anagers rated 17 percent of the SBIR projects as providing more benefits than other agency research projects, 52 percent providing the same benefits, and 31 percent providing fewer benefits. The positive score was similar to that at DoD and NASA projects. The project managers also were asked about two types of outcomes: (1) com- mercialization; and (2) other noncommercial, perhaps intrinsic, uses. At DoE, the project managers reported that 30 percent of projects were commercialized and that 54 percent had noncommercial/intrinsic use. By comparison, the DoD/NASA project monitors reported 35 percent and 68 percent respectively. Overall, project managers indicated that nearly 60 percent of DoE SBIR projects showed a com- mercial or intrinsic use or both. At DoE, only a few projects were identified as receiving Phase III funding from the agency (13 percent). None of the yes responses further indicated that the product of the SBIR award had been directly procured by the agency. 5.3.5 Project Ownership Participation in topic generation is one indication of whether a project man- ager would claim “ownership” of projects resulting from that topic. For about 70 percent of DoE projects, the project manager indicated involvement in the generation of the topic that led to SBIR award; 58 percent reported involvement with the technology before Phase I began, with another 30 percent reporting involvement after Phase I but before Phase II began. Overall, we can define an “ownership group” as those project managers who had a potential stake in the project, as demonstrated either by involvement in topic generation or involvement with the project before the Phase I proposal. For DoE, more than three-fourths of the survey responses had project managers that could be defined as being in the ownership group. All but two DoE project managers reported a technical role in the project. In addition, about one-fifth reported a financial role. It is likely that the financial role was related to such decisions as (1) determining whether the costs of the project were appropriate, (2) deciding whether to fund all or part of the Phase II proposal, and (3) advising the contracting officer on whether sufficient work had been accomplished to justify payments. Only 7 percent reported having a role with respect to commercialization assistance. Finally the project managers were asked if they or others played a role in the project’s ability to obtain Phase III funding. Most DoE project managers (81 per- cent) did not know enough about this subject to answer the question. 

78 Sbir AT THE DEPARTMENT OF ENERGY 5.4 Capitalizing on Program Flexibility 5.4.1 Balancing Commercialization and Mission Orientation The scoring system for SBIR at DoE is intentionally set up to provide flex- ibility to DoE technical programs with different priorities. A program that empha- sizes basic research—for example, Basic Energy Sciences or High Energy and Nuclear Physics—can select proposals for funding that do not receive a strong endorsement on the impact criterion (provided that the criterion is not rated as “having reservations”). Similarly, a program that emphasizes the application of technology—for example, Fossil Energy or Energy Efficiency and Renewable Energy—can select proposals that do not receive a strong endorsement with respect to scientific quality.10 For both of the examples, however, the proposal would have to receive a strong endorsement with respect to the other two criteria, in order to be considered for funding. While the quality of all proposals must be high to be recommended for award, the technical programs now have the free- dom to select any of their candidates for funding for award, and are not forced to select proposals in order of score; i.e., program priorities can now be used as an important factor in award selection. Flexibility also allows the technical programs to develop their own strate- gies for utilizing the capabilities of the small business performers. For example, interviews with staff indicate that the High Energy and Nuclear Physics (HENP) program employs a strategy of using SBIR to develop instrumentation to support the large scale facilities it manages. This strategy exists because they believe that this is the one place where small businesses can contribute—most of HENP research is big science, usually collaborations among large numbers of physicists worldwide, and therefore not amenable to small businesses. The Office of Bio- logical and Environmental Research (BER) also employs SBIR for instrumenta- tion development, frequently soliciting proposals in this area and adopting new measurement tools and methods developed by SBIR participants. DoE technical staff capitalizes on program flexibility in other ways that seek to make the most of complementarities between research programs. Basic Energy Sciences (within the Office of Science), for example, recruits Energy Efficiency and Renewable Energy (EE) personnel to write topics of mutual interest (i.e., topics that tend to have a basic research slant, yet are applicable to EE’s more applied interests). BES puts up their SBIR money; EE puts up the labor to write the topics and manage the proposals. Both programs benefit: BES partners with the applied programs (a specific goal of the program) and EE is able to fund additional SBIR projects. 10In contrast, under the scoring system originally in place for the SBIR program, proposals that received mid-range scores on any criterion were highly unlikely to be funded. 

AGENCY MISSION 79 5.4.2 Internal Reallocation of Topics Among Programs The mandate from Congress is for 2.5 percent of the federal R&D to be allocated to small businesses as part of the SBIR program. Each of the agencies has the freedom to implement this program as appropriate to their ­mission and structure. DoE has since the mid to late 1990s allocated the SBIR award funds on a prorated basis according to the funding of each of its 12 internal program areas. However, some program areas have generated more applications than others. The chance of winning a Phase I award thus varied greatly between programs, as can be seen in the 2005 Award Rates (see Table 5-1). For example, a Phase I SBIR application in the Nuclear Physics program area had a 53 percent probabil- ity of success, while that of an application in Energy Efficiency and Renewable Energy was only 8 percent. On average, 19 percent of DoE SBIR applications in 2005 were successful. These differences in award rates have already started to drive internal r ­ eallocation of funds. Some Basic Energy Sciences topics have been reallocated to Energy Efficiency and Renewable Energy. This internal reallocation occurs case by case, and is negotiated between programs. Some programs also share topics that jointly apply to both programs. However, it seems possible that this might be an area worth further examination by DoE as it seems to ensure that the best quality proposals are funded. TABLE 5-1  2005 SBIR/STTR Award Rates Number Award Number Number of of Phase I Rate Program Area of Topics Applications Awards (%) Fossil energy 7 247 29 12 Advanced scientific computing research 3 47 9 19 Basic energy sciences 9 247 56 23 Biological and environmental research 6 182 47 26 Environmental management 0 0 0 0 Nuclear physics 4 47 25 53 High energy physics 5 111 46 41 Fusion energy sciences 3 80 18 23 Nuclear energy 1 11 3 27 Energy efficiency and renewable energy 6 470 38 8 Nonproliferation and national security 3 61 11 18 Electric transmission and distribution 2 48 8 17 TOTALS 49 1,551 290 19 SOURCE: Department of Energy SBIR program Web site.