Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 21
An Assessment of the SBIR Program at the National Science Foundation 2 Findings and Recommendations NATIONAL RESEARCH COUNCIL (NRC) STUDY FINDINGS The National Science Foundation’s (NSF) Small Business Innovation Research (SBIR) program is adding to the storehouse of public scientific and technological knowledge. The program contributes in many important ways, for example, by: Generating Knowledge. The study showed that projects were yielding a variety of knowledge outputs. Contributions to knowledge are embodied in data, scientific and engineering publications, patents and licenses of patents, presentations, analytical models, algorithms, new research equipment, reference samples, prototype products and processes, spinoff companies, “human capital” (greater know-how, expertise, etc.), and new capability for further innovative activity. Publications and patenting activity occurred with considerable frequency.1 Research quality is difficult to measure, and the value of knowledge created takes time to manifest and has been shown to be highly variable among projects. It can be concluded, however, that the NSF’s SBIR program in its statements of purpose, goals, and criteria—hence, intentions—consistently emphasizes research quality and knowledge creation. Furthermore, the program’s peer-review selection process appears to have integrity in applying the research-merit criterion. Nothing was found by 1 The NRC Phase II Survey reported averages of 1.66 scientific publications and 0.67 patents per surveyed project. The underlying distribution of patents and publications reported is skewed, with some companies reporting none and some reporting relatively high numbers.
OCR for page 22
An Assessment of the SBIR Program at the National Science Foundation the study suggestive of shortcomings in research quality or knowledge creation. (This point is elaborated upon in Chapter 7, Section 7.1.1.) Creating and Disseminating Intellectual Capital. A short-run result of the program’s technological innovation is knowledge outputs. These are important in meeting the NSF’s goal of funding research that leads to broader impact, because they provide paths by which others may use the program’s knowledge gains to achieve additional benefits. There is evidence that the program is producing the kind of knowledge outputs that are typically associated with innovation, such as patents, copyrights, and publications. Extensive licensing activities of Phase II awardees attested to the fact that useful intellectual capital has been created and disseminated. For example, the NRC Phase II Survey showed 20 percent of Phase II projects reporting they had reached licensing agreements with U.S. companies and investors, and 21 percent reporting they had ongoing negotiations with U.S. companies and investors on licensing agreements. (For more detail, see Chapter 7, including Table 7.2-2.) Building Networks with Universities. Both the NRC surveys and the case studies showed extensive networking between NSF SBIR-funded projects and universities. University faculty and students used the NSF SBIR program to establish businesses, start projects, and work on projects. University staff and faculty often assisted with proposal preparation, provided facilities and equipment, and made ongoing contributions to the intellectual capital underpinning company innovations. Faculty also often served as proposal reviewers. (For more details, see Chapter 7, including Table 7.2-5, and also the case studies in Appendix D.) Moving Technology from Universities Toward the Market. The NSF SBIR program has facilitated transfer of technology from universities. Fourteen percent of the NRC Phase II Survey projects were based on technology originally developed at a university by a project participant. Five percent of Phase II Survey projects were based on technology licensed from a university. (This effect is discussed further in Chapter 7, including Table 7.2-5.) Broadening the Scope and Speed of Research. The NSF’s SBIR program has enabled firms to broaden their research scope and to accelerate research. Two-thirds of participating firms reported they “definitely” or “probably” would not have undertaken their projects without an NSF SBIR grant. For those who “definitely” or “probably” would have undertaken the project, 84 percent reported that the project would have been
OCR for page 23
An Assessment of the SBIR Program at the National Science Foundation narrower in scope without the NSF SBIR grant. Most also said their projects would have been delayed without the NSF SBIR grant. (For more details, see Table 5.2-8.) Testing Ideas and Building Capacity. The case studies provided concrete examples of how the NSF’s SBIR program has enabled small businesses to: test creative ideas; develop new capabilities and build capacity; grow intellectual property; gain additional market credibility; and use that enhanced credibility to attract further investment funding while maintaining control of their firms until technical progress enabled them to form more effective partnerships. (These points are discussed further in Section 5.2.4 and Appendix D.) Conducting High-Risk Research. There is evidence that the projects funded were desirably characterized by high technical risk. Technical failure or difficulties were frequently given as a reason by NSF SBIR grantees for not moving to the Phase II stage and for discontinuing Phase II projects once begun. The risk profile of projects is significant to knowledge creation because overcoming challenging scientific and technical hurdles associated with high technical risk is synonymous with increasing the knowledge base. (For more on the technical riskiness of projects, see Sections 5.2.2 and 5.2.3, and Table 5.2-9.) Increasing the Program’s Knowledge Outputs May Be Possible. It appears that the NSF’s SBIR program tends to represent “broader impact” as synonymous with “direct commercial sales” and may not place sufficient emphasis on the encouragement of publishing and other modes of knowledge dissemination by the funded projects. It is recognized that encouragement of knowledge dissemination must occur within the constraints of companies needing to protect their ability to commercialize. Nonetheless, it appears that the NSF’s SBIR program may be able to increase knowledge outputs from the program by signaling to grantees that it places a high value on knowledge outputs, such as publications, papers, presentations, and patents, in addition to commercial outputs. At the same time, it appears that the program is producing knowledge outputs and therefore has the opportunity to compile data on knowledge outputs comprehensively and systematically as part of an improved data collection plan. (See Sections 7.1.2 and 7.2.) NSF SBIR program funding is closely aligned with the NSF’s broader mission and is contributing broadly to federal research and development procurement needs. The program serves as a means of:
OCR for page 24
An Assessment of the SBIR Program at the National Science Foundation Promoting Scientific Progress. Congress created the NSF in 1950 “to promote the progress of science; to advance the national health, prosperity, and welfare; [and] to secure the national defense.” While grants to university researchers continue to be the most common tool the NSF uses to carry out its mission, the SBIR program is another useful tool for NSF to advance science and the nation’s health, prosperity, welfare, and defense. (For more discussion of this point, see Section 4.1.) Helping the Transition of Technology from Universities Toward the Market. NSF SBIR grants further the agency’s academic goals and activities by aiding the transfer of technology out of universities and by helping academic researchers build the small firms that convert their ideas and inventions into commercial products. (These effects are discussed further in Sections 6.3 and 7.2.6 and are illustrated by example in Appendix D.) Meeting Agency Procurement Needs. The NSF SBIR program helps meet the procurement needs of federal agencies. The NRC Phase II Survey found that sales of NSF Phase II–funded technologies go to multiple markets with broad and diversified customer bases. For example, while 57 percent of sales by NSF Phase II survey participants consisted of new and improved products for domestic civilian markets, 20 percent of sales were meeting federal agency procurement needs. (See Section 6.3.2 and Table 6.3-1 for more details.) The NSF’s SBIR program supports a diverse array of small businesses. At the same time, there is a need to improve the analysis of the success rates of woman- and minority-owned businesses, with the goal of increasing their participation in the program.2 Diversity. Small businesses participating in the NSF’s SBIR program are diverse across a number of dimensions. Businesses funded by the NSF are diverse in terms of their age and size, in the technologies they are pursuing, in their form of ownership, in their origin and past history, in their business strategies, in their geographical locations, in their market-sector orientation, in their progress toward commercial success, and in a host of other ways. NSF’s program provides a big tent under which small, innovative businesses across the country can and do develop. (The case studies in Appendix D illustrate the considerable diversity among SBIR-funded firms.) 2 There are inconsistencies in the use of designations. The terminology used here follows that of the Small Business Administration’s description of the SBIR program, which uses the terms “women-owned” and “minority-owned” small businesses. (See SBA.gov/sbir/indexwhatwedo.html.) However, for purposes of this study, “minority-owned” and “disadvantaged” are used interchangeably, although it is recognized that the terms are not interchangeable in federal contracting.
OCR for page 25
An Assessment of the SBIR Program at the National Science Foundation Continuous Influx of Firms New to the Program. The program has demonstrated a continuous influx of firms new to the program. From 1996 through 2003, the period examined, more than half of each year’s SBIR award recipients had not previously received an NSF SBIR grant, with the exception of one year. In the most recent year examined, 2003, more than 60 percent of grantees were new to the SBIR program. As the program continuously approves applications from these newly proposing firms, the population of small firms helped by the SBIR program grows. (Table 4.2-8 provides additional details on newly proposing firms.) A Commercial Enabler for Small Firms. Although few individual SBIR projects lead directly to “home runs” in the commercial sense, many SBIR projects funded by federal agencies contribute to agency missions and to the broader economy. Both the NRC Phase II Survey and the set of case studies showed that small firms believe that the NSF’s SBIR program helped them enter commercial markets. Role in Project Initiation. The SBIR awards play a key role in initiating commercially oriented research. When asked if their companies would have undertaken the projects had there been no SBIR grant, approximately two-thirds of respondents answered either probably not (43 percent) or definitely not (24 percent). (See Table 5.2-8.) Awards as Enablers. Case study firms called the NSF’s SBIR program an “enabler” and a “lifeline.” They said the SBIR awards enabled them to: start a company; survive while they were trying to refine their ideas to the point where they could attract private money; build capacity; develop new technologies and improve, renew, and lower the cost of existing technologies; build on and add other sources of funding; grow an intellectual property portfolio essential to commercial success; retain control of their firms; and pursue multiple paths to commercialization, including contract research. (For details, see Sections 5.2.2, 5.2.4, and 5.2.5, Table 5.2-11, and Appendix D.)
OCR for page 26
An Assessment of the SBIR Program at the National Science Foundation Flexibility for Small Companies. Because the NSF generally cannot provide formal financial support into Phase III by contracting with grantees, its SBIR recipients do not have the Phase III funding options often available to recipients of SBIR funding from the procurement agencies.3 On the other hand, the NSF funds a variety of technology areas, which allows firms more leeway in aligning their research with market demand. (For more details, see Section 6.2.) While women and minorities have benefited from the NSF’s SBIR program, both as company owners and as principal investigators, there is room for improvement. Participation Rates of Women and Minorities. Levels of participation by woman- and minority-owned businesses have continued to lag other groups. From 1995 through 2005, woman-owned businesses submitted 12.2 percent of Phase I proposals and received 9.5 percent of Phase I grants. They submitted 8.8 percent of Phase II proposals and received 7.5 percent of Phase II grants. Minority-owned businesses (including minority women) submitted 16 percent of Phase I proposals and received 13.5 percent of Phase I grants. They submitted 12.9 percent of Phase II proposals and received 13.7 percent of Phase II grants. Woman and minorities also participated in projects as principal investigators, with 21 percent of Phase II projects surveyed reporting either a woman, a minority, or a minority woman as the principal investigator. (For elaboration on participation by women and minorities as business owners and as principal investigators, see Section 4.2.5.) Room for Improvement. The cause of the lower participation rates is unclear. It may reflect the low representation of women and minorities in high technology firms.4 These lower success rates may also reflect 3 As noted, the NSF’s SBIR program does make available additional support to selected firms through its Phase IIB activities. 4 White males, who comprise 40 percent of the nation’s overall workforce, hold 68 percent of all science, engineering, and technology jobs. In contrast, white women, who comprise 35 percent of the national workforce, hold only 15 percent of these positions, and only 10 percent of the 2 million scientists and engineers in the United States in a recent year were women. African Americans and Hispanics, who comprise almost 21 percent of the American workforce, represent just 6 percent of the science, engineering, and technology workforce. (Congressional Commission on the Advancement of Women and Minorities in Science, Engineering and Technology Development. Findings of the Commission as reported in SSTI Weekly Digest, April 6, 2001; and U.S. Bureau of Labor
OCR for page 27
An Assessment of the SBIR Program at the National Science Foundation obstacles that women and minorities face in pursuing careers in science and engineering.5 Moreover, it seems noteworthy that their participation does not seem to be trending upward despite the recent gains these groups have made in other areas. It is a source of concern that the success rates for woman- and minority-owned firms at Phase I are below those of other firms.6 It is also troubling that the number of NSF SBIR proposals and grants to woman- and minority-owned businesses do not show a positive trend over the last decade when gains by these groups were being made in other areas. There may be room for improved participation in the program for woman- and minority-owned firms. Further analysis is required to determine what steps, if any, are required. More Examination Needed. To understand the causal factors behind these statistics for woman- and minority-owned firms requires further investigation beyond the scope of this study, particularly at the Phase I proposal review stage. Are there systematic shortcomings in the application process that could be addressed through customized outreach and targeted training? Are there problems in proposal review and selection that could be addressed? Finally, are there actions that could proactively encourage greater participation by woman- and minority-owned firms? (See Section 4.2.5.) Statistics.) However, it should be noted that not all minority groups have low representation in science, engineering, and technology fields relative to their representation in the U.S. population Indicative of shifting representation among minority groups, there was a strong increase during the 1990s in the percentage of doctorate degrees and jobs in science and engineering going to foreign-born individuals, particularly those from India, China, and the Philippines. (National Science Foundation, Science and Engineering Indicators 2006, “The U.S. S&E Labor Force,” Arlington, VA: National Science Foundation, 2006.) By like token, the representation in the NSF’s SBIR program is uneven among different minority groups. 5 Academics represent an important future pool of applicants, firm founders, principal investigators, and consultants. Recent research shows that owing to the low number of women in senior research positions in many leading academic science departments, few women have the chance to lead a spinout. “Underrepresentation of female academic staff in science research is the dominant (but not the only) factor to explain low entrepreneurial rates amongst female scientists.” See Peter Rosa and Alison Dawson, “Gender and the commercialization of university science: academic founders of spinout companies,” Entrepreneurship & Regional Development 18(4):341–366. 6 The lower success rates of woman- and minority-owned firms in obtaining Phase I grants were found by tests for statistical significance to be highly significant. The probabilities of getting the t-test results if there were no differences in success rates of woman- and minority-owned firms versus all firms at the Phase I stage was 0.000 at a level of significance of 0.05. The results of the tests of differences at Phase II were weaker; at the Phase II stage, the probabilities of getting the t-test results if there were no differences in success rates were less than 0.23 for both woman- and minority-owned firms. (For more details, see Section 4.2.5, particularly Figures 4.2-12, 4.2-13 and 4.2-18.)
OCR for page 28
An Assessment of the SBIR Program at the National Science Foundation FIGURE 2-1 Comparative Success Rates for Woman-owned versus All Applicants in Having Their Phase I Proposals Approved, 1994–2005. SOURCE: Developed from data provided by the NSF SBIR program. NOTE: Differences are statistically significant. Although the agency itself normally does not acquire the results of SBIR-funded projects, NSF projects do achieve commercialization.7,8 NSF SBIR Commercialization Outcomes: The NRC Phase I and Phase II Surveys sought to identify whether some form of commercialization activity had occurred for projects and firms contacted. In interpreting these data, it should be noted that they include firms that may have had very limited sales (e.g., several units for test purposes); only a relatively small proportion of firms have high commercial success rates, as described below. 7 The lack of a procurement function tends to limit commercialization at the NSF in comparison with procurement agencies, such as the Defense Department. 8 The NSF’s SBIR program has recently emphasized its goal to “commercialize research.” Interest in promoting commercialization appeared to intensify with the centralization of the program within a single office. This emphasis on commercialization contrasts with the concentration of most of NSF’s resources on academic research. The SBIR program holds that the small share of resources that goes to business should emphasize research for business purposes. The committee finds this position entirely reasonable and applauds the program’s efforts to foster commercialization.
OCR for page 29
An Assessment of the SBIR Program at the National Science Foundation Of the Phase I projects that did not result in a direct follow-on Phase II grant, nearly half reported sales, expected sales, or some other form of progress. Of the Phase II projects included in the survey: Twenty-two percent reported that the referenced projects had resulted in commercial products, processes, or services in use and were still active. (Recall that this figure includes projects for which sales were quite limited.) Nineteen percent reported that commercialization was under way. Ten percent reported that they achieved sales, licensing, or additional funding before discontinuing the projects. (Note again that this figure includes projects for which sales were quite small.) Twenty-eight percent indicated that they were continuing technology development after completing the Phase II research. Altogether, 69 percent of respondents for Phase II projects indicated some form of progress toward either commercial or continuing technological development. (For more details, see Sections 5.2.1, 5.2.2, and 5.2.3 and Figures 5.2-3 and 5.2-4.) A Small Percentage of Projects Account for Most Successes. As is typical for other private and public technology innovation programs, a relatively few projects have accounted for the majority of sales and licensing revenue from NSF SBIR recipients.9 (For details, see Section 5.2.) Outcomes Are Skewed. A smaller percentage of projects will likely achieve large growth and huge sales revenues, i.e., be commercial “home runs.”10 These patterns are similar to those found in other private and public investments in high-risk advanced technology investments, where many research projects must be seeded to yield a few commercial “home runs.”11 9 Similar results were revealed in previous National Research Council assessments of early-stage innovation awards. See National Research Council, The Advanced Technology Program: Assessing Outcomes, Charles W. Wessner, ed., Washington, DC: National Academy Press, 2001. See also National Research Council, The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative, Charles W. Wessner, ed., Washington, DC: National Academy Press, 2000. 10 Among the 162 projects surveyed, just 8 projects—each of which had $2.3 million or more in sales—accounted for over half the total reported sales dollars for the surveyed projects. The project with the highest reported amount had $4.8 million in sales. Similarly, the results for sales by licensees of those survey projects’ technologies were highly skewed by a single licensee that accounted for over half the total licensee sales dollars, amounting to $200 million or more in licensee sales. 11 As with investments by angel investors or venture capitalists, SBIR awards result in highly con
OCR for page 30
An Assessment of the SBIR Program at the National Science Foundation Meanwhile, many small successes together comprise a potentially important component of the nation’s innovative capability. Balancing Technological Innovation and Success in Commercialization. In reviewing findings on commercialization, it is important to keep in mind the inherent tension between the program’s goal to stimulate advanced technological innovation and its goal to increase private-sector commercialization. Some would argue that a research program achieving a high percentage of commercial successes would suggest a failure to attack sufficiently challenging technical problems. It is also important to keep in mind that much can be learned from technical failure; unsuccessful projects can suggest what paths not to follow and identify hurdles that remain to be overcome. Commercialization as a Core Goal. In recent Senate testimony, a senior program official affirmed that the program’s focus has become “the commercialization of research.”12 To increase commercialization, the program has taken several steps that represent a change from earlier practice. The program now: requires a business plan as part of Phase II proposals; provides commercialization assistance training and networking assistance to award-recipient firms; centrated sales, with a few awards accounting for a very large share of the overall sales generated by the program. These are appropriate referent groups, though not an appropriate group for direct comparison, not least because SBIR awards often occur earlier in the technology development cycle than where venture funds normally invest. Nonetheless, returns on venture funding tend to show the same high skew that characterizes commercial returns on the SBIR awards. See John H. Cochrane, “The Risk and Return of Venture Capital,” Journal of Financial Economics 75(1) 2005:3–52. Drawing on the VentureOne database Cochrane plots a histogram of net venture capital returns on investments that “shows an extraordinary skewness of returns. Most returns are modest, but there is a long right tail of extraordinary good returns. Fifteen percent of the firms that go public or are acquired give a return greater than 1,000 percent! It is also interesting how many modest returns there are. About 15 percent of returns are less than 0, and 35 percent are less than 100 percent. An IPO or acquisition is not a guarantee of a huge return. In fact, the modal or ‘most probable’ outcome is about a 25 percent return.” See also Paul A. Gompers and Josh Lerner, “Risk and Reward in Private Equity Investments: The Challenge of Performance Assessment,” Journal of Private Equity 1 (Winter 1977):5–12. Steven D. Carden and Olive Darragh, “A Halo for Angel Investors,” The McKinsey Quarterly 1 (2004), also show a similar skew in the distribution of returns for venture capital portfolios. 12 Testimony of Joseph Hennessey, “The Small Business Innovation Research Program: Opening Doors to New Technology,” before the House Committee on Small Business’s Subcommittee on Workforce, Empowerment and Government Programs, November 8, 2005.
OCR for page 31
An Assessment of the SBIR Program at the National Science Foundation uses Phase IIB awards to encourage firms to seek third-party financing and to provide a market test of commercial potential; designs its solicitation topics to be more in line with industry interests; and uses more reviewers with business experience in the proposal selection process. (For elaboration on NSF’s commercialization efforts, see Chapter 5, and for more on the Phase IIB program, see Section 8.5.)13 Commercialization May Be Further Increased. For example, the NRC Phase II Survey found that Phase II projects were often discontinued for market-related reasons; in particular, companies cancelled projects because market demand was too small or the product/process/service was not competitive. It may be possible for the program to reduce the percentage of projects going forward into Phase II with insufficient market potential by continuing and intensifying improvements to outreach, commercialization assistance, and project review/selection. This would require additional resources for commercialization assistance, which are constrained at 1992 levels. (See Chapter 5, particularly Table 5.2-9, and Chapter 8, Section 8.4.4.) Firms that have won multiple SBIR awards were a point of study. The Single Award Assumption. Some companies that have received numerous SBIR awards have been criticized as SBIR “mills.” Implicit in this criticism is the assumption that one or a few awards should be sufficient for a company to launch a product and become “independent” from the SBIR program. In some cases, initial awards may, in fact, be sufficient to attract private investors and enable a company to develop exclusively through private funding. Developing Technologies. For companies pursuing technologies with limited markets (e.g., NASA or the DoD) or, in the case of the NSF, technologies that may have broad applications, a single set of SBIR awards may not be sufficient to develop the technology to the point that it is attractive for procurement (by another agency) or for funding by private capital markets.14 13 As noted above, the program recently has taken steps to increase the rate of commercialization. At the time of this study’s surveys, however, there had been insufficient time to determine if these steps have increased rates of commercialization. The results of such efforts may begin to show up in post-survey data. 14 For companies pursuing multiple technology areas, multiple grants may entail no overlap. For example, Luna Innovations, Inc., has won SBIRs from multiple agencies for multiple technology
OCR for page 32
An Assessment of the SBIR Program at the National Science Foundation Role of Multiple Awards. The case study results support the hypotheses that a single grant is seldom adequate to bring an idea to market readiness and that multiple funding sources, multiple projects, and multiple technologies are frequently needed for success in commercialization. At the same time, single grants can and do have a significant impact, opening a line of inquiry or revealing an opportunity. Diversity of Firm Objectives. As noted above, the NSF program attracts quite a large percentage of firms new to the program each year—some 60 percent in 2003. Nonetheless, not all firms are new participants. Some firms that complete work successfully win again. Indeed, some companies utilize the program as a means of exploring technologies that they believe will have commercial promise, while others seek to resolve research questions of relevance to government missions. Some of these companies are essentially contract research organizations, where fulfilling the terms of SBIR solicitations is an important part of their business model, one that also meets government needs. Addressing Agency Missions. With regard to frequent award winners, a key issue is the research quality and cost. Small companies often have lower overheads than some other alternative sources of research, e.g., national laboratories and large companies that often receive many federal contracts. Companies that are providing quality research that meets agency solicitation requirements should not be subject to an arbitrary limit. Discontinue Nonperforming Frequent Award Winners. However, in cases where a company has received a number of past SBIR grants for which the research has been of poor quality or the company has otherwise failed to carry through on its proposal, agency management should (and does) reduce or eliminate awards to the company unless and until there is compelling evidence favoring additional grants. In the case where a company is good at writing proposals but poor at execution, this may require an intervention after the peer review process. (According to NSF’s program descriptions, it applies such intervention as necessary.) areas, including multiple Phase I, Phase II, and Phase IIB SBIR grants from the NSF. For every $1 of SBIR funding received, Luna has generated at least $2 in non-SBIR funding. It has spun off six companies—each specializing in a different technical field ranging from optical devices to life sciences to advanced materials. It has used the SBIR program as a funding source to get a number of ideas off the ground and move them along a market-focused pipeline. Founded in the early 1990s, Luna was listed on the NASDAQ in June 2006.
OCR for page 33
An Assessment of the SBIR Program at the National Science Foundation The NSF’s SBIR program is well run, largely effective at achieving program goals, and has benefited from its strong, centralized management and talented program managers.15 Effective and Respected Management. The NSF’s SBIR program benefits from a centralized, well-coordinated management team that is widely seen as effective in managing the program. (See Sections 4.3 and 8.9.1.) Skilled Program Managers. Program managers have firsthand industry experience. Case study firms spoke positively of the program’s management team, describing the program managers as “highly motivated,” “knowledgeable,” “flexible,” and “effective.” The firms described the turnover among program staff as lower than at other SBIR programs and characterized the lower turnover as an advantage to program participants.16 (See Section 8.9.1 and Appendix D.) Funding Across a Broad Spectrum of Technology Solicitations. Case study firms spoke of the program’s willingness to fund a variety of technologies, including manufacturing projects and those with long development cycles, such as advanced materials. The firms noted that the program’s relatively broad topic definitions allow them leeway to align their projects with in-house strengths and market demands. (For more details, see Sections 5.2.4 and Appendix D.) Responsive to External Critiques. The NSF’s SBIR program regularly receives and responds to external advice. It receives external oversight from its Advisory Committee and from its Committee of Visitors (COV), which meets triannually. Continuous program improvements, reflecting management’s focus on program improvement, were observed in the course of the study. (For more details, see Section 8.13.2.) Active Outreach and Training. The NSF’s SBIR program runs active outreach, training, mentoring, and networking programs, and it has taken a leadership role in recent national SBIR conferences. The program has sponsored SBIR spring and fall national conferences in recent years. It participates in state SBIR workshops and has scheduled one of its national conferences each year in an EPSCoR state (a state participating in the “Experimental 15 The effectiveness of this management approach at the NSF should not necessarily be extrapolated to other agencies or departments (e.g., Defense) that have programs ten times larger with widely distributed budgets. 16 More recently the program has experienced staff turnover.
OCR for page 34
An Assessment of the SBIR Program at the National Science Foundation Program to Stimulate Competitive Research and Institutional Development Awards”).17 (For more details, see Section 8.2.) In 1999, the NSF’s SBIR program began a commercialization assistance workshop for Phase I awardees, and since 2001 the program has contracted for a Commercialization Planning Assistance program for Phase I awardees using Dawnbreaker, Inc., and (later) Foresight Science & Technology, Inc. (For more details, see Section 8.4.) After implementing its commercialization assistance activities, program officials reported improvements in Phase II proposals. In 2005 the program began sponsoring participation by Phase II award recipients in an annual “Opportunity Forum.” (For more details, see Sections 8.2 and 8.4.) Notwithstanding the program’s strong program management, the Committee identified a number of areas that may benefit from change. There are several issues that should be addressed to increase performance. Inflexible and Inadequate Management Budget. Resources for program administration and commercialization assistance are inadequate. Program administrative budgets reportedly do not vary with changes in SBIR funding. The administrative budget has fallen both in constant dollars and as a percentage of the total program size. Increasingly, the program has had to do more with less, which over time may negatively impact performance and appears to be a constraint in initiating new initiatives, including some mentioned below, e.g., assessment. (See Sections 8.12 and 8.4.4.) Limited Assessment Efforts. The program’s past efforts at assessment, while commendable, were limited because the evaluations have been ad hoc rather than systematic; internal and/or limited in availability; constrained by firms not meeting post-project reporting goals; and hampered by ad hoc definitions of “success.” The program’s inadequate budget for evaluation is one reason for the limited amount of assessment. Inadequate Attention to, or Discouragement of, Knowledge Outputs. There appear opportunities for the program to provide a more consistent message to program participants about the importance of knowledge outputs, including scientific publications, to encourage knowledge outputs, 17 Currently, EPSCoR is aimed at 25 states, Puerto Rico, and the U.S. Virgin Islands—jurisdictions that have historically received lesser amounts of federal R&D funding.
OCR for page 35
An Assessment of the SBIR Program at the National Science Foundation and to compile a comprehensive set of indicators of knowledge creation and dissemination for evaluation purposes. Limited Electronic Data Collection. Systematic and reliable compilation of data in electronic form useful for program management and evaluation has, until recently, been limited. (See Sections 4.2.2 and 8.8.) “Topic Gap” and Need for Transparent, Bottom-up Topic Selection Process. Despite the NSF’s broad solicitation topics, there may be a “topic gap” that arises when firms must wait until their topic is featured in a solicitation. According to program staff, firms may have to wait up to 18 months for a particular topic area to be part of an NSF solicitation. Long intervals between topic solicitations may prevent some firms from taking advantage of time-limited opportunities. Furthermore, the selection of topics may lack transparency and represent more of a top-down than a bottom-up process. (See Section 8.1, particularly 8.1.3.) Controversial Application of “Additional Factors.” The program’s use of “additional factors” for proposal selection lacks transparency. These factors are mentioned briefly in program guidance to applicants, but there is little or no explanation as to how they are used. While some additional factors, such as “past commercialization efforts by the firm when previous grants exist,” appear valid, others seem more problematic. Some interviewed firms were particularly troubled by the additional factor, “excessive concentration of grants in one firm.” While the NSF’s desire to avoid excessive concentration of grants may have merit, firms claim that they do not know how “excessive concentration” is defined. Moreover, this factor is applied after firms have incurred the cost of submitting a proposal, and several surveyed firms believe that it is applied unevenly among firms. It should be noted that the firms interviewed did not necessarily object to a limit on how many grants they could receive from the NSF, but they did want to know in advance if there is a limit, what the limit is if there is one, and how the rule will be evenly applied across firms. (For additional discussion, see Section 8.3.3.) Limits on the Number of Proposals per Company. The NSF limits firms to four proposals per solicitation. Larger eligible companies that have more areas of research and/or more scientists feel constrained by this limit. Given that a goal of the SBIR program is to promote innovative activity among small firms, and given that the act of developing proposals has itself been found to stimulate innovation, the program’s emphasis should be to encourage firms to develop SBIR proposals. On the other hand, the program’s administrative budget is limited, there appears to be
OCR for page 36
An Assessment of the SBIR Program at the National Science Foundation no shortage of grant-worthy proposals with the limit in effect, and the limit works against higher concentrations of grants to particular firms. (For elaboration, see Section 8.1.3 and Appendix D.) NRC STUDY RECOMMENDATIONS Recognizing the accomplishments already achieved, the Committee recommends that the NSF reinforce its efforts to improve commercialization. Greater efforts, including outreach and better data collection, should be made to raise participation by women and minorities. Better documentation and evaluation are also recommended in order to improve program output and to facilitate program management. Consideration should be given to the provision of additional program funds to realize these objectives. Above all, every effort should be made to preserve program flexibility. Preserve program flexibility. SBIR agencies and departments, the Small Business Administration, and the Congress should seek to ensure that program adjustments and refinements do not reduce the flexibility with which the program is administered. The SBIR program is effective across the agencies partly because a “one-size-fits-all” approach has not been imposed. This flexible approach should be continued, subject to appropriate monitoring, across the departments and agencies in order to adapt the program to their evolving needs and to improve its operation and output. Continue to increase private-sector commercialization of innovations derived from federal research and development. Continue to promote Phase IIB awards, refining the tool as experience suggests and raising the number and amount of these awards as third-party funding permits. Increase use of technically competent reviewers with strong technical expertise and strong business understanding for both Phase I and Phase II selection. Increase support for commercialization assistance as resources permit. While recognizing the difficulty of assessing markets for new-to-the-market and disruptive innovations, reduce as possible the substantial share of projects for which it is determined during Phase I that perceived market demand is inadequate for the grantee to continue pursuing the innovation even if Phase I is technically successful.
OCR for page 37
An Assessment of the SBIR Program at the National Science Foundation Improve participation and success of women and minorities. Encourage Participation. Develop targeted outreach to improve the participation rates of woman- and minority-owned firms and strategies to improve their success rates based on causal factors determined by analysis of past proposals (see item IIa.) and feedback from the affected groups. Improve Data Collection and Analysis. Arrange for an independent analysis of a sample of past proposals from woman- and minority-owned firms and from other firms (to serve as a control group) for the purpose of identifying specific factors accounting for the lower success rates of woman- and minority-owned firms as compared with other firms in having their Phase I proposals granted. Extend Outreach to Younger Women and Minorities. The NSF should immediately encourage and solicit women and underrepresented minorities working at small firms to apply as principal investigators (PIs) and senior co-investigators (Co-Is) for SBIR awards and track their success rates. Encourage Emerging Talent. The number of women and, to a lesser extent, minorities graduating with advanced scientific and engineering degrees has been increasing significantly over the past decade, especially in the biomedical sciences. This means that many of the women and minority scientists and engineers with the advanced degrees usually necessary to compete effectively in the SBIR program are relatively young and may not yet have arrived at the point in their careers where they own their own companies. However, they may well be ready to serve as principal investigators and/or senior co-investigators on SBIR projects. Track Success Rates. The Committee also strongly encourages the NSF to gather the data that would track women and minority principal investigators and to ensure that SBIR is an effective road to opportunity for these PIs as well as for women- and minority-owned firms. The success rates of women and minority PIs and Co-Is is the traditional measure of their participation in the non-SBIR research grants funded by nonmission research agencies like NIH and the NSF. It is also a very appropriate measure of women and minority participation in the SBIR program. After all, experience as a PI or Co-I on a successful SBIR program may well give a woman or minority scientist or engineer the personal confidence and standing with agency program officers that encourage them to found their own SBIR firms.
OCR for page 38
An Assessment of the SBIR Program at the National Science Foundation Regular, rigorous program evaluation is essential for quality program management and accountability. Accordingly, the NSF program management should give greater attention and resources to the systematic evaluation of the program supported by reliable data and should seek to make the program as responsive as possible to the needs of small company applicants. To this end, the program should undertake a systematic compilation of performance measures for program activities, outputs, and outcomes measured against specific goals. In order for the SBIR program to be agile and responsive to applicants, performance measures should include how quickly proposal reviews take place and funds are moved to grantee companies. The evaluation should incorporate both internal efforts and arms-length external efforts, and evaluation studies should be made publicly available in an annual report. Ensure that solicitation topics are broadly defined, that the definition process is bottom-up, and take steps to ensure the necessary flexibility to permit firms to receive relatively prompt access to Phase I solicitations. Consider adding an “other category” within a solicitation, if needed, to keep it open to all promising topics. Alternatively, explore the feasibility of a pilot program designed specifically to decrease the delay between solicitations for given topics. Recognizing that transparency in project selection procedures is important to the perceived fairness of a public funding program, the NSF’s SBIR program should adopt the following steps: Revisit the “additional factors” affecting award decisions and decide which ones are worth keeping, and then make these factors explicit in program guidance, in terms of their definitions and how, when, and by whom they are to be judged. In particular, clarify additional factor #3—“Excessive concentration of grants in one firm”—in terms of how it is defined and applied, and ensure that firms can determine prior to proposing if they will be affected. NSF management should ensure that all project selection factors are clear, transparent, and evenly applied.
OCR for page 39
An Assessment of the SBIR Program at the National Science Foundation The use of quotas to reduce multiple winners should be reconsidered. In the case of multiple award winners who qualify in terms of the selection criteria, the acceptance/rejection decision should be based on their performance on past grants rather than on the number of grants received. Firms able to provide quality solutions to solicitations should not be excluded, a priori, from the program except on clear and transparent criteria (e.g., quality of research and/or commercialization performance). Avoid imposing quotas on applications unless there are compelling reasons to do so. Specifically, remove the limit of four on the number of proposals per solicitation per firm. Such limitations run the risk of limiting innovative ideas and of unnecessarily restricting the opportunity among prospective principal investigators in the larger eligible small companies. In order to continue to attract new entrants and to help avoid concentrations in existing companies, the NSF should continue its focus on outreach to ensure a high rate of proposals from firms new to the program. Increase management funding for SBIR. To enhance program utilization, management, and evaluation (as described above), consideration should be given to the provision of additional program funds. There are three ways by which this might be achieved: Additional funds might be allocated internally, within the existing NSF budget, with reference to the management funding provided for comparable NSF programs, keeping in mind the special requirements of SBIR applications, solicitations, evaluations, selection, monitoring, reporting, outreach, commercialization services, site visits, and other functions related to the normal and effective operation of the program. Funds might be drawn from the existing set-aside for the program to carry out these activities.18 The set-aside for the program, currently at 2.5 percent of external research budgets, might be increased, with the goal of providing additional resources to maximize the program’s return to the nation.19 18 Under current legislation, funds drawn from the SBIR “set-aside” cannot be used for these purposes. They are almost exclusively allocated for awards. 19 Each of these options has its advantages and disadvantages. For the most part, the departments, institutes, and agencies responsible for the SBIR program have not proved willing or able to make additional management funds available. Without direction from the Congress, they are unlikely to do so. With regard to drawing funds from the program for evaluation and management, current legislation does not permit this and would have to be modified; therefore the Congress has clearly intended program funds to be for awards only. The third option, involving a modest increase to the program, would also require legislative action and would perhaps be more easily achievable in the event of an
OCR for page 40
An Assessment of the SBIR Program at the National Science Foundation overall increase in the program. In any case, the Committee envisages an increase of the “set-aside” of perhaps 0.03 to 0.05 percent on the order of $35 million to $40 million per year or, roughly, double what the Navy currently makes available to manage and augment its program. In the latter case (0.05 percent), this would bring the program “set-aside” to 2.55 percent, providing modest resources to assess and manage a program that is approaching an annual spending of some $2 billion. Whatever modality adopted by the Congress, without additional resources the Committee’s call for improved management, data collection, experimentation, and evaluation may prove moot.