Cover Image

HARDBACK
$73.75



View/Hide Left Panel

Evaluation of the Department of Defense Small Business Innovation Research Program and Fast Track Initiative: A Balanced Approach*

Robert B. Archibald and David H. Finifter

The College of William and Mary

EXECUTIVE SUMMARY

The purpose of this paper is to evaluate the Department of Defense (DoD) Small Business Innovation Research (SBIR) program and the Fast Track initiative. The SBIR program has grown and changed in its brief history, and the Fast Track Initiative is a recent innovation in the program. The SBIR program has a dual nature. It is designed to invigorate federal R&D efforts by expanding the participation of small business in the largest federal research programs and to increase the commercialization of innovative products and services that result from federal R&D efforts. Given the dual nature of the program it is important that evaluation of the program consider both the research and the commercial outcomes.

The SBIR Program—The SBIR Program was started in 1982 as a set-aside program for small business. It diverts a portion of the extramural research or research and development budgets of eleven federal agencies to fund the awards. The SBIR program has grown over time. Currently, each agency must devote not less than 2.5 percent of its extramural research and development budget to its SBIR program, representing a considerable increase from the original 0.2 percent of this budget. The SBIR awards are divided into three types. Phase I awards are small and intended to determine the scientific and technical merit and feasibility of the ideas. These awards are very competitive; overall there is an average of one award for every six to seven Phase I proposals. Phase II awards are designed to

*

Prepared for the May 5th SBIR Symposium sponsored by the Board on Science, Technology, and Economic Policy of the National Research Council, National Academy of Sciences (NAS).



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 211
The Small Business Innovation Research Program: AN ASSESSMENT OF THE DEPARTMENT OF DEFENSE FAST TRACK INITIATIVE Evaluation of the Department of Defense Small Business Innovation Research Program and Fast Track Initiative: A Balanced Approach* Robert B. Archibald and David H. Finifter The College of William and Mary EXECUTIVE SUMMARY The purpose of this paper is to evaluate the Department of Defense (DoD) Small Business Innovation Research (SBIR) program and the Fast Track initiative. The SBIR program has grown and changed in its brief history, and the Fast Track Initiative is a recent innovation in the program. The SBIR program has a dual nature. It is designed to invigorate federal R&D efforts by expanding the participation of small business in the largest federal research programs and to increase the commercialization of innovative products and services that result from federal R&D efforts. Given the dual nature of the program it is important that evaluation of the program consider both the research and the commercial outcomes. The SBIR Program—The SBIR Program was started in 1982 as a set-aside program for small business. It diverts a portion of the extramural research or research and development budgets of eleven federal agencies to fund the awards. The SBIR program has grown over time. Currently, each agency must devote not less than 2.5 percent of its extramural research and development budget to its SBIR program, representing a considerable increase from the original 0.2 percent of this budget. The SBIR awards are divided into three types. Phase I awards are small and intended to determine the scientific and technical merit and feasibility of the ideas. These awards are very competitive; overall there is an average of one award for every six to seven Phase I proposals. Phase II awards are designed to * Prepared for the May 5th SBIR Symposium sponsored by the Board on Science, Technology, and Economic Policy of the National Research Council, National Academy of Sciences (NAS).

OCR for page 211
The Small Business Innovation Research Program: AN ASSESSMENT OF THE DEPARTMENT OF DEFENSE FAST TRACK INITIATIVE enable firms to do the research proposed in Phase I. Current Phase I awards normally have a maximum of $100,000 and Phase II awards have a maximum of $750,000, although these ceilings may be exceeded at the discretion of the program managers. Phase III awards do not actually involve SBIR funds. The terminology is used to describe situations in which actual production for agency use is funded through the continuation of a successful SBIR project. The rationale for aid to small businesses generally relies on claims of discrimination in capital markets. The discrimination in capital markets is frequently called statistical discrimination. The fact that statistics indicate small businesses typically have higher default rates than large businesses is used to deny the loan to a small business applicant. Statistical discrimination leads to social inefficiency because small firms have less access to capital than they would if information were perfect. When it created the SBIR program, Congress was also concerned with discrimination in government procurement. The concern here is with research productivity, i.e., a budget that relies heavily on large business results in lower productivity for research and development than would a budget that increases small business participation. Much of the mission-essential research carried out by government agencies is in areas that are highly innovative. In the industries in which small firms may have an advantage, it is sensible technology policy for the government to target some of its R&D funds on small firms. The Fast Track Initiative—The 1996 Fast Track Initiative of DoD represents a continuation of the shift in emphasis in the SBIR award process toward commercial success. Under Fast Track firms with Phase I contracts which can interest outside investors in committing funds to further the development of the project increase their chances of obtaining Phase II funding and are eligible for bridge funding between Phase I and Phase II. The increase in the importance of commercial success is clear. A firm that does a piece of research that increases knowledge of a government laboratory provides a useful service to the government by aiding the ongoing research of the laboratory. Such a firm will, however, not be able to attract outside investors, so it will not be eligible for Fast Track. Fast Track is reserved for firms that are likely to be commercial successes through producing a product or service that can be directly sold, or whose product or service holds sufficient commercial promise that outside investors are willing to invest in its further development. Fast Track fits a particular model of small business. It is designed for a small business that has technical expertise and a desire to use that technical expertise to develop a product or service that it can sell, either in the commercial marketplace or as a government contractor, or both. This firm has no particular desire to be a small business. There are two other types of firms that participate in the SBIR program. First, some successful small businesses have no desire to be big busi nesses. These firms are not growth-oriented. Some of these were created started

OCR for page 211
The Small Business Innovation Research Program: AN ASSESSMENT OF THE DEPARTMENT OF DEFENSE FAST TRACK INITIATIVE by refugees from laboratories in large businesses and others by researchers who got their start in university or government laboratories. The founders of these firms were frustrated by the bureaucracy in the large business or some part of their academic or government job and became convinced to start or join a small business. Second, there are growth-oriented firms with products that are not yet commercially viable. These firms would like to expand and become big businesses, but the product or service they are developing is too far from its final state to be interesting to outside investors. Survey of Technical Monitors—Our independent contribution to the evaluation is measurement of the quality and usefulness to the federal government of the research conducted by the SBIR firm, i.e., we provide measures of research success. The individuals who are in the best position to assess the quality and usefulness of SBIR research to the government are government scientists and engineers who monitor the research progress. Our evaluation takes the form of a survey of the technical monitors of the SBIR Phase II contracts. The technical monitors, called either Contracting Officer’s Technical Representative (COTR) or Technical Points of Contact (TPOC), are the DoD scientists or engineers who served as the liaison between the small business and the agency that awarded the contract. One innovative aspect of the survey is that it was conducted via e-mail to make it practical and encourage a high response rate. Because it was designed for e-mail distribution, the survey was as short as possible while capturing key questions. We had a very high response rate for e-mails that were received by a COTR, although we were unable to find good e-mail addresses for a large number of COTRs. The database contains 379 SBIR Phase II contracts. The sample was drawn systematically. It includes all of the 1996 Fast Track projects and all of the BMDO Matching projects from 1992 to 1996 as well as a matched sample of regular Phase II projects. As such it does not represent a random sample of the entire DoD SBIR program. We have responses covering 195 (51.5%) of the contracts in the full sample. Our overall response rate for successful contacts, 78.9%, is very high. The survey had two parts and covered five major areas. The first part of the survey focused on individual SBIR projects, measuring (1) research quality, (2) usefulness of the research, and (3) mission benefits of the research. The second part of the survey focused on the SBIR program, measuring (4) overall quality of SBIR proposals, and (5) impressions of Fast Track. Quality of Research—We asked two questions about the quality of research. One question asked for a rating of the quality of the research for the SBIR Phase II and the second asked for a rating of the quality of other research. For our sample, the mean value for the difference between the rating given to the research quality of the SBIR project and the rating given to other research is .025 with a standard deviation of 2.366.

OCR for page 211
The Small Business Innovation Research Program: AN ASSESSMENT OF THE DEPARTMENT OF DEFENSE FAST TRACK INITIATIVE Usefulness of the Research—To discover whether or not the SBIR research was useful we ask survey respondents to indicate whether or not the research has affected the way that research is conducted in their unit/office. 25.94% indicated “No, the knowledge generated by this SBIR contract has had no impact on the other research we conduct or sponsor. The other 74.06% chose one of the responses that started with “Yes, this project produced results that have been useful to us, . . .” Mission Benefits—We asked the respondents to compare the mission benefits per dollar on the SBIR project with the average benefits per dollar on other research. 31.58% indicated that the SBIR project yielded more benefits per dollar, 42.11% indicated it yielded the same benefits per dollar, and 26.54% indicated it yielded less benefits per dollar. Overall Quality of Proposals—We asked each COTR about whether he or she was satisfied with the quality of the SBIR proposals his or her office received. 63.91% indicated that they had more good proposals than they could fund, 25.18% indicated that they had the same number of good proposals as they could fund, and 10.91% indicated that they had fewer good proposals than they could fund. Fast Track—We asked those COTRs who had been involved with a Fast Track about its effectiveness. 66.67% indicated that it was more effective than the normal program. 21.57% indicated that it was the same as the normal program, and 11.76% indicated that it was less effective than the normal program. In summary, the DoD SBIR program received a very favorable evaluation from the technical monitors. The quality of the research in the SBIR program is indistinguishable from the quality of other research. Regarding usefulness of the research, over 74% of the responses indicated that the research was useful. Just over 73% of the responses indicated that a dollar spent on the SBIR project yielded the same or more benefits per dollar than other research efforts. Over 81% of the respondents said that they had the same, or more, good SBIR proposals than they could fund. And the Fast Track Initiative was rated as more effective than the normal SBIR program by two thirds of the respondents. In the analysis of the survey results we divided the sample among the Fast Track projects, the BMDO Matching projects and the comparison group projects. Perhaps surprisingly, the Fast Track projects did very well in these comparisons. Despite the fact that the Fast Tract Initiative was designed to enhance the commercial success rate of its projects, according to our results the research outcomes of the Fast Track projects compared favorably to those in the other two samples. Balanced Evaluation—The fundamental assumption of our approach to evaluation is that the two goals of the SBIR program should be given equal weights. Under this approach, projects that produce commercial success should count as successful SBIR projects, and projects that produce useful research for

OCR for page 211
The Small Business Innovation Research Program: AN ASSESSMENT OF THE DEPARTMENT OF DEFENSE FAST TRACK INITIATIVE the contracting agency should count as successful SBIR projects. A proper evaluation of the program has to consider these two possible avenues to success for each project. This approach corrects a fundamental flaw in current evaluations, the fact that, exclusively, these evaluations have been separate evaluations of either commercial success or research success. To accomplish a balanced evaluation we need information on both research outcomes and commercial outcomes. To obtain such data, we combine the results from our survey of COTRs with the results from a survey of firms conducted by Peter Cahill of BRTRC. The survey of firms was designed primarily to provide information on commercial outcomes. The linked survey responses therefore allow us to measure interactions between the research and commercial goals of the SBIR program. A majority of the projects for which we have information from technical monitors is contained in the data set of firm responses. There are 124 observations in the linked data set. For given definitions of research success and commercial success, we can classify projects into one of four outcomes. Table 1 illustrates our evaluation strategy. Table 1 is the basis for our evaluation of the SBIR program. There are several measures of research and commercial outcomes, and for some measures it is not obvious where one should draw the line between success and failure. For this reason, there are many ways one could classify projects. In the text we provide several examples using different definitions of the two types of success. Our major finding is that in almost all of the examples we considered, commercial and research success were statistically independent. This means we did not find a trade off between commercial success and research success. Also, we found that projects in the Fast Track Initiative generally did very well both in measures of research outcomes and measures of commercial outcomes. Summary and Conclusions—There are three major conclusions in this paper. The balanced evaluation we outline is the appropriate way to evaluate the SBIR program. Almost all of the previous evaluations have only considered commercial success. The quality of the research in the SBIR program has received scant attention. This paper provides the first evaluation that looks simultaneously at the two goals of the program. TABLE 1 Classifications of SBIR Projects   Research Failure Research Success Commercial Failure Group N—Neither a research nor a commercial success Group R—Research successes that are not commercial successes Commercial Success Group C—Commercial successes that are not research successes Group B—Both a commercial success and a research success

OCR for page 211
The Small Business Innovation Research Program: AN ASSESSMENT OF THE DEPARTMENT OF DEFENSE FAST TRACK INITIATIVE Our survey of technical monitors indicates that the DoD SBIR produces high quality research that in useful to the overall research effort of the department. Also, this high quality research does not appear to conflict with the SBIR goal of increasing the commercialization of the results of government R&D. Contrary to our initial expectations, projects in the DoD Fast Track Initiative appear to produce research that is as good if not better than the research in the normal DoD SBIR program.

OCR for page 211
The Small Business Innovation Research Program: AN ASSESSMENT OF THE DEPARTMENT OF DEFENSE FAST TRACK INITIATIVE INTRODUCTION The purpose of this paper is to evaluate the Department of Defense (DoD) Small Business Innovation Research (SBIR) program and the Fast Track Initiative. The SBIR program has grown and changed in its brief history, and the Fast Track Initiative is a recent innovation in the program. The results of surveys of technical monitors and SBIR awardees, which we report here in detail, indicate that these technical monitors have a strongly positive assessment of the Fast Track Initiative and the Fast Track firms have generally been very successful. The focus of this paper is on the appropriate evaluation of the SBIR program and the Fast Track Initiative. The SBIR program has a dual nature. It is designed to invigorate federal research and development (R&D) efforts by expanding the participation of small business in the largest federal research programs and to increase the commercialization of innovative products and services that result from federal R&D efforts. Given the dual nature of the program, it is important that evaluation of the program consider both the research and the commercial outcomes. Seven additional sections follow. In the second section, we give a brief description of the SBIR program and discuss its history. In the third section, we provide an outline of the theoretical rationale for the program. We introduce the Fast Track Initiative in the fourth section and explain its role in the context of the SBIR program. In the fifth section, we present evidence from a survey of technical monitors of SBIR contracts. In the sixth section, we provide an analysis of the results of the survey of technical monitors, focusing on research outcomes for the DoD SBIR program and the Fast Track Initiative. In the seventh section, we combine the survey of technical monitors with a survey of firms. The combined survey allows us to do a balanced evaluation of the DoD SBIR program and the Fast Track Initiative, including information on both goals of the SBIR program. The final section contains a summary and our conclusions. DESCRIPTION OF THE SBIR PROGRAM The SBIR program was started in 1982 as a set-aside program for small business. It diverts a portion of the extramural research or R& D budgets of 11 federal agencies to fund the awards. The SBIR program has grown over time. Currently, each agency must devote at least 2.5 percent of its extramural research R&D budget to its SBIR program, representing a considerable increase from the original 0.2 percent of this budget. The SBIR awards are divided into three types. Phase I awards are small and intended to determine the scientific and technical merit and feasibility of the ideas. These awards are very competitive; overall there is an average of one award for every six to seven Phase I proposals. Phase II awards are designed to enable firms to do the research proposed in Phase I. They are granted to roughly one-third of the winners of Phase I awards. Phase III awards do not actually involve

OCR for page 211
The Small Business Innovation Research Program: AN ASSESSMENT OF THE DEPARTMENT OF DEFENSE FAST TRACK INITIATIVE SBIR funds, but this terminology is used to describe situations in which actual production for agency use is funded through the continuation of a successful SBIR project. On occasion, private commercial funding also is described as Phase III funding. Current Phase I awards normally have a maximum of $100,000 and Phase II awards have a maximum $750,000, although these ceilings may be exceeded at the discretion of the program managers. Each agency establishes its own policies and priorities regarding the categories of projects funded by its SBIR program, receives and evaluates proposals, selects awardees for SBIR funds, and makes the appropriate payments. The Small Business Administration (SBA) sets the schedule for the solicitation of proposals and, along with the Office of Science and Technology Policy, receives annual reports from each agency that runs an SBIR program. To evaluate the SBIR program, it is important to identify the legislative intent for and goals of the program. The Small Business Innovation Development Act of 1982 was the result of a recommendation of the first White House Conference on Small Business in January 1980. The delegates to this conference voted to recommend the bill that authorized the creation of the SBIR program. There were several reasons this piece of legislation found support in the conference and eventually in the Congress. First, evidence suggested that small businesses had been having difficulty obtaining funds in general and had a declining share of federal R&D contracts. Second, several well-publicized studies indicated that small businesses were a very important source of job growth1, and the recessions of the early 1980s created a supportive climate for any proposal that could claim job creation potential. Third, a successful SBIR pilot project established by the National Science Foundation demonstrated that the program was feasible. From the start, Congress had two goals for the SBIR program. A report of the Senate Committee on Small Business makes this clear. The purpose of the bill is twofold: to more effectively meet R&D needs brought on by the utilization of small innovative firms (which have been consistently shown to be the most prolific sources of new technologies) and to attract private capital to commercialize the results of the Federal research. (U.S. Congress, 1981, p. 1) The two primary goals for the program are also clear from the legislation. The 1982 act that created the SBIR program listed the following objectives of the program: to stimulate technological innovation; to use small business to meet Federal research and development needs; to foster and encourage participation by minority and disadvantaged persons in technological innovation; and to increase private sector commercialization of innovations derived from federal research and development. (96 Stat. 217) 1   See, for example, David Birch (1981).

OCR for page 211
The Small Business Innovation Research Program: AN ASSESSMENT OF THE DEPARTMENT OF DEFENSE FAST TRACK INITIATIVE The act was originally scheduled to expire on October 1, 1988, but during fiscal year 1986, Congress enacted legislation extending the law through September 1993 (99 Stat. 443). In 1992 the SBIR program was reauthorized. The Small Business Innovation Reauthorization Act of 1992 both raised the percentage of research expenditures dedicated to the SBIR program and increased the importance of the goal of commercializing SBIR projects. The goal of commercialization moved from being listed fourth in 1982 to second in 1992. This change in the ordering of the goals was purposeful and was reflected in important ways in the language describing the selection process after 1992. Specifically, the original language describing Phase I was: a first phase for determining, insofar as possible, the scientific and technical merit and feasibility of ideas submitted pursuant to SBIR program solicitations: (96 Stat. 218) This language was amended as follows (the added language is underlined): a first phase for determining, insofar as possible, the scientific and technical merit and feasibility of ideas that appear to have commercial potential as described in subparagraph (B)(ii). submitted pursuant to SBIR program solicitations: (106 Stat. 4250) For Phase II the change is more dramatic. The original language was: a second phase to further develop the proposed ideas to meet the particular program needs, the awarding of which shall take into consideration the scientific and technical merit and feasibility evidenced by the first phase and where two or more proposals are evaluated as being of approximately equal scientific and technical merit and feasibility, special consideration shall be given to those proposals that have demonstrated third phase, nonFederal capital commitments; (96 Stat. 218) This was changed to: a second phase, to further develop proposals which meet particular program needs, in which awards shall be made based on the scientific and technical merit and feasibility of the proposals as evidenced by the first phase considering, among other things, the proposal ’s commercial potential, as evidenced by: the small business concern’s record of successfully commercializing SBIR or other research; the existence of second phase funding commitments from private sector or non-SBIR funding sources; the existence of third phase, follow-on commitments for the subject of the research; and the presence of other indicators of the commercial potential of the idea. (106 Stat. 4251)

OCR for page 211
The Small Business Innovation Research Program: AN ASSESSMENT OF THE DEPARTMENT OF DEFENSE FAST TRACK INITIATIVE These are clear mandates to change the selection processes by increasing the importance of commercial potential. Under the 1982 legislation, ties between projects deemed to be of equal scientific and technical merit could be broken in favor of projects that were more likely to be commercially successful. The likelihood of commercialization was clearly a secondary concern. This was changed with the 1992 legislation, which placed commercialization on an equal footing with scientific and technical merit. The 1996 Fast Track Initiative of DoD represents a continuation of the shift in emphasis in the SBIR award process toward commercial success. Under Fast Track, firms with Phase I contracts that can interest outside investors in committing funds to further the development of the project increase their chances of obtaining Phase II funding and are eligible for bridge funding between Phase I and Phase II. The increase in the importance of commercial success is clear. A firm that does a piece of research that increases the knowledge base of a government laboratory provides a useful service to the government by aiding the ongoing research of the laboratory. However, such a firm will not be able to attract outside investors, and so it will not be eligible for Fast Track. Fast Track is reserved for firms that are likely to be commercial successes through production of a product or service that can be directly sold, or whose product or service holds sufficient commercial promise that outside investors are willing to invest in its further development. RATIONALE FOR THE SBIR PROGRAM An important feature of the SBIR program is that it is a set-aside. It does not result from new monies appropriated by Congress. It results from Congress mandating that agencies engaged in research target a portion of their existing funds for research projects carried out by small businesses. Because of this, the SBIR program represents a redirection of R&D spending, not an expansion. R&D provides a public good and, for that reason, it is a sensible public expenditure.2 The question raised by the existence of the SBIR program is: Why does the government want to increase the involvement of small business in this activity? We think of the SBIR program as addressing two failures: a market failure and a government procurement failure. First, the SBIR program is one of several programs designed to help small businesses. The loan and loan guarantee programs administered by the SBA provide other examples. These programs address a failure in credit markets. Second, the SBIR program seeks to correct deficiencies in federal procurement practices that lead to an excessive reliance on large businesses for federal R&D. Evaluations of the SBIR program should focus on how well the program corrects these two failures. 2   See Griliches (1992) for a review of the literature on this point.

OCR for page 211
The Small Business Innovation Research Program: AN ASSESSMENT OF THE DEPARTMENT OF DEFENSE FAST TRACK INITIATIVE The SBIR Program as Aid to Small Businesses The rationale for aid to small businesses generally relies on claims of discrimination in capital markets. Frequently called statistical discrimination,3 it occurs when, for example, a lender decides to make a loan to a large business, not to a small business, simply on the basis of size; that is, the two businesses are otherwise equally creditworthy. The fact that statistics indicate that small businesses typically have higher default rates than large businesses is used to deny the loan to a small business applicant.4 This kind of decision is cost-effective for the lender, but it is easy to see why government remedies are appealing. Statistical discrimination leads to social inefficiency because small firms have less access to capital than they would if information were perfect. This rationale for support of small business is generally recognized. One important question the SBIR program raises is: Is there anything about R& D activity or high-technology firms that exacerbates these problems? The answer to this last question is probably “yes.” R&D is a much riskier investment than are many other types of investments that a firm might undertake. Because of this riskiness, R&D investment requires a much higher rate of return in capital markets than financing an expansion in current capacity. As Bronwyn Hall points out “asymmetric information between firms and investors implies that, to fund projects about which they do not have full information, investors will demand a ‘lemons’ premium in the form of a higher rate of return.” (1993, p. 290). This lemon effect for R&D investment applies to all firms, but it is very likely to be an even bigger problem for small firms. Josh Lerner (1999) explains how the SBIR program helps small firms to overcome these difficulties. He argues that a small firm that is successful in the competition for SBIR funds sends a signal of its capability to outside investors. In essence, to outside investors this certification indicates that winners of SBIR awards are less likely to produce R&D lemons. One could argue that the special problems that small businesses have obtaining R&D funds might have been overcome by creating a special category of SBA loans or loan guarantees to support R&D. If the argument put forth by Lerner is correct, the current SBIR program is probably a superior alternative. The program gives small firms a chance to enter a competition in which federal technical experts judge their R&D proposals. A small firm’s successful competition in this arena signals that it is likely to be better than other small businesses in the commercial arena as well. 3   See Arrow (1973) for an early discussion of statistical discrimination. 4   Stiglitz and Weiss (1981) demonstrate why financial institutions’ response to this problem is to deny credit rather than simply to raise the interest rates they charge on loans to risky borrowers.

OCR for page 211
The Small Business Innovation Research Program: AN ASSESSMENT OF THE DEPARTMENT OF DEFENSE FAST TRACK INITIATIVE TABLE 6 Classification of Research and Commercial Success Using Usefulness of Research and Sales   Research Success Commercial Success Yes (74.5%) No (24.5%) Yes (37.9%) Group B, 30 (25.8%) Group C, 14 (12.1%) No (62.1%) Group R, 56.5 (48.7%) Group N, 15.5 (13.4%) the various classifications? Table 7 presents information on this question. The table shows the prevalence of projects with various characteristics in the linked sample and for the projects in the two extreme categories: Group B, both a research and a commercial success, and Group N, neither a research nor a commercial success. The entries in the table are percentages of projects in the particular classification that are part of a particular group. For example, 25.4% of the full sample are Fast Track projects, but 19.4% of the Category B projects are Fast Track projects. Though none of the differences in proportions are statistically significant, the results for Fast Track are mixed. Fast Track projects are slightly underrepresented in Category B, which does not reflect well on Fast Track, and very underrepresented in Category N, which reflects well on Fast Track. If it is more important to avoid Category N than it is to hit Category B, as seems sensible, then these are essentially positive results for Fast Track. The results for BMDO Matching projects appear to be uniformly inferior. Overall the Comparison group firms seem better in this comparison. Firms with more than three prior Phase II awards appear to be inferior by these measures also. In contrast, having fewer than three prior Phase II awards results in a statistically significant lower proportion of firms in Category N. This is the only statistically significant difference for this table. Finally, there does not appear to be a clear pattern for results based upon the number of employees in the firm. TABLE 7 Analysis of Example 1 Classifications by Groups Variable Full Sample (%) Group B (%) Group N (%) Fast Track 25.4 19.4 6.7 BMDO Matching 25.4 16.1 53.3 Comparison 49.2 64.5 40.0 Prior Phase II ≤ 3 81.1 90.3 60.0 Prior Phase II > 3 18.9 9.7 40.0 Employees ≤ 10 58.0 54.8 50.0 Employees > 10 42.0 45.2 50.0

OCR for page 211
The Small Business Innovation Research Program: AN ASSESSMENT OF THE DEPARTMENT OF DEFENSE FAST TRACK INITIATIVE Example 2: Sales and Mission Benefits In this example we keep the same definition of commercial success and change the definition of research success. We base research success on Question 6 in the COTR survey: In comparison to a dollar spent in your research unit/office on other R&D projects, did a dollar spent on this SBIR project yield more/the same/less benefits for your agency’s mission than the average dollar spent on other contracts sponsored by your research unit/office. We define research success as yielding more benefits than the average on other contracts, a very strict definition of research success. In this case, the test of the hypothesis of independence yields a chi-square statistic of 1.209, which is not statistically significant. Table 9 presents the analysis of these categories by the groups we used above. In these comparisons, again Fast Track is underrepresented in Group B, but this time Fast Track is also overrepresented in Group N as well, the uniformly poor rating. The BMDO Matching sample has mixed results, overrepresented in Group B and slightly underrepresented in Group N. None of the results for Fast Track or the BMDO Matching is statistically significant. Prior Phase II experience is again not related to favorable outcomes according to these findings. Again, firms with three or fewer prior Phase II awards were significantly less likely statistically to be in Category N. The results for firm size indicate that small firms are more than proportionately related to being both a commercial and a research success. For firm size, the result that firms with 10 or fewer employees were more likely to be in Group B is statistically significant. The two examples illustrate that this type of analysis depends critically on the way one defines success. In Table 6, we defined research success in terms of whether the research in the SBIR provided results that were useful to the agency—a minimal notion of success. In Table 8, the research success was reserved for projects that provided more benefits to the agency mission than the average contract—a much stricter notion of research success. In Table 8, with the lower hurdle for research success, over 86% of the topics registered some type of success. In Table 8, with the higher hurdle for research success, just over 56% of the projects registered some type of success. TABLE 8 Classification of Research and Commercial Success Using Mission Benefits and Sales   Research Success Commercial Success Yes (33.9.5%) No (66.1%) Yes (38.1%) Group B, 18 (15.3%) Group C, 14 (22.9%) No (61.9%) Group R, 22 (18.6%) Group N, 51 (43.2%)

OCR for page 211
The Small Business Innovation Research Program: AN ASSESSMENT OF THE DEPARTMENT OF DEFENSE FAST TRACK INITIATIVE TABLE 9 Analysis of Example 2 Classifications by Groups Variable Full Sample (%) Group B (%) Group N (%) Fast Track 25.4 18.2 31.4 BMDO Matching 25.4 31.8 24.5 Comparison 49.2 50.0 55.9 Prior Phase II ≤ 3 81.1 95.4 72.5 Prior Phase II > 3 18.9 4.6 27.5 Employees ≤ 10 58.0 77.3 52.1 Employees > 10 42.0 22.7 47.9 SUMMARY AND CONCLUSIONS Our summary and conclusions come under four headings: (1) conclusions about appropriate evaluation of the SBIR program, (2) results for the general DoD SBIR program, (3) results for the Fast Track Initiative, and (4) suggestions for future research. Appropriate Evaluation of the SBIR Program The SBIR program is difficult to evaluate. Most previous evaluations have focused on commercial success and have been based on surveys of firms. The quality of the research in the SBIR program has received little attention. This paper provides the first evaluation that looks simultaneously at the two goals of the program. There are three methodological points we want to emphasize. The SBIR program has two goals: to increase the participation of small business in federal R&D and to increase the commercialization of innovations developed as a result of federal R&D. An appropriate evaluation has to account for both of the program goals. E-mail is a good medium to evaluate outcomes of the research. The technical monitors—the DoD scientists and engineers who worked with the firms—were generally quite willing to respond to an e-mail survey. Our analysis of the responses from the survey of technical monitors and the survey of firms illustrates the appropriate methodology for evaluating a program with two clear goals. One objective of any agency ’s SBIR program should be to minimize the projects that are neither a research nor a commercial success. Our methodology illustrates the first attempt to evaluate progress for this objective. Results Regarding General DoD SBIR Program Several of our results provide an evaluation of the performance of the overall DoD SBIR program as a research program. In general, the results are very positive. There are five findings we would like to highlight:

OCR for page 211
The Small Business Innovation Research Program: AN ASSESSMENT OF THE DEPARTMENT OF DEFENSE FAST TRACK INITIATIVE Based on our survey responses, the quality of the research conducted in the SBIR program is indistinguishable from other research conducted by or sponsored by DoD. Based on our survey responses, a large majority (74.04 percent) of the SBIR projects were rated as having produced results that were useful for the ongoing research effort at DoD. Based on our survey responses, a large majority (73.66 percent) of the SBIR projects were rated as having produced the same or more mission benefits per dollar than other R&D projects of DoD. Based on our survey, a substantial majority (61.14 percent) of the technical monitors indicated that they had more good SBIR proposals than they could fund, and only a small proportion (11.07 percent) indicated that they had fewer good SBIR proposals than they could fund. Using multivariate analysis, we found no clear evidence that the research success of the DoD SBIR program could be improved based upon information available to program managers prior to the award of the contract. The SBIR program is being considered for reauthorization in the next Congress. One of the questions that will come up in that debate is whether it is sensible to expand the percentage of the R&D budget set aside for the SBIR program. This is a complex issue. Our results indicate that, at the current size, the quality of DoD SBIR research is high and there are more good proposals than can be funded. Although these results are not sufficient to suggest an expansion of the program, they probably represent necessary conditions. Results Regarding the Fast Track Initiative The sample we used was designed to study the Fast Track Initiative. We gathered information on Fast Track in two ways. We asked the technical monitors who had experience with the Fast Track Initiative for a direct evaluation and, in the analysis of our results, we separated the sample to see if performance by Fast Track projects differed from performance by the comparison groups. The Fast Track Initiative generally is viewed very favorably and the results for Fast Track projects are impressive. Two-thirds of the technical monitors who had experience with a Fast Track project rated Fast Track as more effective in advancing the research program in their unit. Only 11.76 percent of these respondents rated Fast Track as less effective in advancing the research program in their unit. Though the differences were not statistically significant, the quality rating for Fast Track projects indicated an advantage for Fast Track compared to other projects. Though the differences were not statistically significant, the percentage of Fast Track projects rated as being useful for the research effort was higher than for non-Fast Track projects.

OCR for page 211
The Small Business Innovation Research Program: AN ASSESSMENT OF THE DEPARTMENT OF DEFENSE FAST TRACK INITIATIVE These results are impressive because the Fast Track Initiative was designed to improve the commercial success, not the research success, of the DoD SBIR program. Given the current size of Fast Track—5 percent of the DoD SBIR program—there clearly is not a trade-off between commercial and research success. Although perhaps unexpected, the lack of any reduction in research success may result from the fact that Fast Track projects have larger research budgets. One of the clear advantages of the Fast Track Initiative is that it attracts funds to the federal R&D effort. Our results suggest this has been beneficial for research. Issues Deserving Further Research One limitation of the research reported in this paper is the sample design. A correct evaluation of the DoD SBIR program requires a random sample. The linked sample provides a very interesting view of outcomes in the SBIR program. The objective of program managers should be to reduce the number of failures, that is, projects that are neither research nor commercial successes. It would be very important to search for the correlates of projects that are classified in this category from a random sample of SBIR projects. The preliminary analysis with our sample uncovered some interesting findings. In particular, it appears that firms with three or fewer prior Phase II contracts are underrepresented in the failures whereas firms with more than three prior Phase II contracts are overrepresented. It would be very interesting to see if such findings hold for a random sample. ACKNOWLEDGMENTS We would like to thank Joe Heyman, Thayer Sheets, and Bob Yang of NASA-Langley Research Center for valuable discussions about the workings of the SBIR program, Charles Wessner and John Horrigan (NAS) for useful comments on an earlier version of this paper, Peter Cahill (BRTRC) for help with data, and public policy graduate students Stephen Bowman, Jennifer Jebo, Randy Rosso, and Keith Wandtke for valuable research assistance. The above-listed individuals are not responsible for any shortcomings of the paper. The views expressed are those of the authors and do not represent the view of the College of William and Mary. Please do not cite without permission. REFERENCES Acs, Zoltan J., and David B. Audretsch. 1990. Innovation and Small Firms. Cambridge, MA: MIT Press. Arrow, Kenneth. 1973. “The theory of discrimination,” pp. 3-31 in Discrimination in Labor Markets, Orley Ashenfelter and Albert Rees, eds. Princeton, NJ: Princeton University Press. Birch, David. 1981. “Who creates jobs,” The Public Interest 65(Fall):3-14.

OCR for page 211
The Small Business Innovation Research Program: AN ASSESSMENT OF THE DEPARTMENT OF DEFENSE FAST TRACK INITIATIVE Freeman, Chris, and Luc Soete. 1997. The Economics of Industrial Innovation. Cambridge, MA: MIT Press. Griliches, Zvi. 1992. “The search for R&D spillovers,” Scandinavian Journal of Economics 94:S29-S47. Hall, Bronwyn H. 1993. “Industrial research during the 1980s: Did the rate of return fall? ” Brookings Papers: Microeconomics 2:289-343. Hamberg, Dan, 1963. “Invention in the industrial research laboratory,” Journal of Political Economy (April):95-115. Lerner, Josh. 1999. “The government as venture capitalist: The long-run effects of the SBIR program,” Journal of Business. July, v. 72 3, pp. 285-297. Mervis, Jeffrey D. 1996. “A $1 Billion ‘Tax’ on R&D Funds,” Science 272:942-944. OECD (Organization for Economic Cooperation and Development. 1982. Innovation in Small and Medium Firms. Paris: OECD. Roberts, Edward B. 1968. “Entrepreneurship and technology,” Research Management (July):249-266. Rubenstein, A. H. 1958. Problems Financing New Research-Based Enterprises in New England. Boston MA: Federal Reserve Bank. SBA (Small Business Administration). 1992. Results of Three-Year Commercialization Study of the SBIR program . Washington, D.C.: U.S. Government Printing Office. Scherer, F. M. 1970. Industrial Market Structure and Economic Performance. New York: Rand McNally College Publishing. Stiglitz, J. E., and A. Weiss. 1981. “Credit rationing in markets with incomplete information,” American Economic Review 71:393-409. U.S. Congress, House. 1992. SBIR and Commercialization: Hearing Before the Subcommittee on Technology and Competitiveness of the House Committee on Science, Space, and Technology, on the Small Business Innovation Research [SBIR] Program (testimony of James A. Block, President of Creare Inc.) pages 356-361. U.S. Congress, Senate Committee on Small Business. 1981. Small Business Research Act of 1981. S.R. 194, 97th Congress. U.S. General Accounting Office. 1989. Federal Research: Assessment of the Small Business Innovation Research Program, GAO/RCED-89-39, Washington, D.C.: GAO. U.S. General Accounting Office 1992. Small Business Innovation Research Program Shows Success But Can Be Strengthened, RCED-92-32. Washington, D.C.: GAO. U. S. General Accounting Office. 1998. Federal Research: Observations on the Small Business Innovation Research Program, RCED-98-132. Washington, D.C.: GAO.

OCR for page 211
The Small Business Innovation Research Program: AN ASSESSMENT OF THE DEPARTMENT OF DEFENSE FAST TRACK INITIATIVE APPENDIX A Means of Variables This table gives the means of the variables for various subsamples of the data. For simplicity these means are for the respondents who answered both Question 5 and Question 6. This causes some of the means to differ slightly from those reported in the text. TABLE A1 COTR Survey Results   Useful Mission Benefit   Quality Q5 % Q6 % Subsample N Q3-Q4 b,c,d a b Total 184 .0511 73 32 42 0 Prior Phase II 86 .2500 79 33 42 1-10 Prior 78 −.0395 69 32 40 >10 Prior 20 −.4000 65 25 50 Women/Min. 47 .2727 66 38 36 1-9 Employees 82 .4494 74 40 33 10-19 Emp 38 .0429 68 26 47 20-50 Emp. 33 −.6613 73 18 52 >50 Emp. 31 −.2419 77 29 48 Fast Track 31 .5862 84 29 48 92 Award 16 −.5625 63 19 56 93 Award 21 .5000 86 33 43 94 Award 30 −.5000 60 20 27 95 Award 33 .0156 73 48 30 96 Award 63 .2756 77 31 49 Computers 31 −.1207 71 26 42 Electronics 83 .3063 70 36 42 Materials 12 −1.6667 58 8 58 Mechanical 9 .6111 78 44 22 Energy Cons. 15 −.2000 73 27 47 Environment 4 .0000 100 50 50 Life Sciences 1 −4.000 100 0 0 Air Force 28 .5357 79 43 32 Army 31 .3214 87 39 32 BMDO 67 .0308 63 25 49 DARPA 33 −.2419 76 33 39 DSWA 4 .6250 75 75 0 Navy 18 −.6470 72 17 50 OSD 3 −.3333 100 0 100 Ownership 77 −.0205 82 32 44

OCR for page 211
The Small Business Innovation Research Program: AN ASSESSMENT OF THE DEPARTMENT OF DEFENSE FAST TRACK INITIATIVE APPENDIX B Multivariate Analyses This appendix presents the correlation and regression results discussed in Section VI. The variable definitions are given below followed by the correlation matrix and the estimated regressions. Because there were several missing values for the topic categories, for each of the three dependent variables, SBIR Quality, Useful, and Benefits, there are two regressions, one using all the variables, and one excluding the variables for topic categories. The regressions for Useful and Benefits are logit regressions. Variable Definition SBIR Quality a rating on a scale of 1 to 10 for the quality of the SBIR project Other Quality a rating on the same scale of the average quality of non SBIR research conducted in the last three years Useful a categorical variable equal to one if the SBIR project’s research results were useful Benefits a categorical variable equal to one if the SBIR project’s benefits to the mission (per dollar) were greater than the benefits to the mission (per dollar) from other projects AF a categorical variable equal to one if the project was an Air Force funded project ARMY a categorical variable equal to one if the project was an Army funded project BMDO a categorical variable equal to one if the project was a BMDO funded project DARPA a categorical variable equal to one if the project was a DARPA funded project DSWA a categorical variable equal to one if the project was a DSWA funded project NAVY a categorical variable equal to one if the project was a Navy funded project OSD a categorical variable equal to one if the project was an OSD project Fast Track a categorical variable equal to one if the project was a Fast Track project BMDO Match a categorical variable equal to one if the project was a BMDO Matching project AGE age of the project computed as 97 minus the fiscal year for the project older projects will have larger values. SIZE size of the SBIR firm measured as the number of employees Experience experience in the SBIR program measured as the number of prior Phase II awards for the firm Ownership a categorical variable for those projects for which the COTR was involved with defining or generating the topic and was involved with the firm before the Phase I proposal COMPUTER a categorical variable equal to one if the topic area was Computers ELECTRON a categorical variable equal to one if the topic area was Electronics MATERIAL a categorical variable equal to one if the topic area was Materials VEHICLES a categorical variable equal to one if the topic area was Mechanical Performance of Vehicles, Weapons, or Facilities ENERGY a categorical variable equal to one if the topic area was Energy Conservation and Use ENVIRON a categorical variable equal to one if the topic area was Environment and Natural Resources LIFE SCIENCES a categorical variable equal to one if the topic area was Life Sciences

OCR for page 211
The Small Business Innovation Research Program: AN ASSESSMENT OF THE DEPARTMENT OF DEFENSE FAST TRACK INITIATIVE TABLE B1 Correlation Matrix   (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (1) SBIR Quality 1.000                   (2) Other Quality .1571 1.000                 (3) Useful .3840 .0095 1.000               (4) Benefits .3731 −.1403 .2533 1.000             (5) AF .0762 −.0503 .0494 .1254 1.000           (6) ARMY .0639 .0487 .1487 .0606 −.1908 1.000         (7) BMDO −.1048 −.1424 −.1763 −.1108 −.3270 −.3270 1.000       (8) DARPA .0244 .1051 .0669 .0391 −.1986 −.1986 −.3403 1.000     (9) DSWA .0815 .0375 .0141 .1408 −.0657 −.0657 −.1125 −.0683 1.000   (10) NAVY −.0681 .0882 −.0296 −.1061 −.1452 −.1452 −.2488 −.1511 −.0500 1.000 (11) OSD −.0116 −.0158 −.0397 −.1143 −.0736 −.0736 −.1262 −.0766 −.0253 −.0560 (12) Fast Track .0967 −.0202 .1156 −.0043 −.1087 .3841 −.1700 .0013 −.0657 −.0445 (13) BMDO Match −.0124 −.0754 −.1998 −.0730 −.2699 −.2699 .8353 −.2809 −.0929 −.2053 (14) AGE −.2291 −.1500 −.0493 −.0561 −.0871 −.3673 .4206 −.0236 .0057 −.0080 (15) SIZE .0362 .0593 −.0315 .0487 .0017 .0842 −.1507 −.1068 .0868 .1263 (16) Experience −.0054 .0030 −.0592 −.0293 .1310 .0019 −.0791 −.0993 .0400 .0400 (17) Ownership .0187 .1119 .1432 .0105 .1978 .1978 −.3723 −.0402 .0210 .1209 (18) COMPUTER −.0202 .0908 −.0837 −.0336 .1134 .0739 −.3167 .0584 −.0697 .1847 (19) ELECTRON .0284 −.1153 −.0313 .0835 −.0599 −.1508 .1832 −.0552 .0159 −.0021 (20) MATERIAL −.1122 .0511 −.1031 −.1425 −.0632 −.1215 .2378 −.0697 −.0418 −.0209 (21) VEHICLES .1016 .0353 .0355 .0638 .1772 −.0999 −.0123 .0309 −.0344 −.0760 (22) ENERGY −.0752 −.0388 .0173 −.0312 −.0220 −.1313 .1092 .0761 −.0452 −.0329 (23) ENVRION .0304 .0375 .0967 .0599 .0368 −.0657 −.1125 .1312 −.0226 .0756 (24) LIFE SCIENCES −.1266 .0399 .0480 −.0505 .1706 −.0326 −.0339 −.0558 −.0112 −.0248

OCR for page 211
The Small Business Innovation Research Program: AN ASSESSMENT OF THE DEPARTMENT OF DEFENSE FAST TRACK INITIATIVE (11) (12) (13) (14) (15) (16) (17) (18) (19) (20) (21) (22) (23) (24)                                                                                                                                                                                                                                                             1.000                           .0183 1.000                         −.1041 −.2699 1.000                       −.1504 −.3897 .3039 1.000                     .1858 −.0589 −.1164 −.0223 1.000                   .1191 −.1469 −.0474 .0233 .5007 1.000                 .0575 .1065 −.3380 −.1786 .1866 .0212 1.000               .0986 .1134 −.2539 −.1656 −.0237 −.0493 .0354 1.000             .0517 .1521 .1149 −.1332 −.0013 .0091 −.0876 −.4171 1.000           –.0469 −.1215 .2110 .3569 −.0063 .1089 −.0260 −.1289 −.2504 1.000         −.0386 −.0306 .0292 .0607 .0494 −.0138 .0062 −.1060 −.2059 −.0636 1.000       −.0507 −.0767 .0832 .1345 −.0977 −.0476 .0622 −.1393 −.2705 −.0836 −.0688 1.000     −.0253 −.0657 −.0929 .1176 −.0898 .0247 .0210 −.0697 −.1353 −.0418 −.0344 −.0452 1.000   −.0126 −.0326 −.0460 .0444 −.0342 −.0288 −.0649 −.0345 −.0671 −.0207 −.0170 −.0224 −.0112 1.000

OCR for page 211
The Small Business Innovation Research Program: AN ASSESSMENT OF THE DEPARTMENT OF DEFENSE FAST TRACK INITIATIVE TABLE B2 Regression Equations   SBIR Quality Useful Benefits Constant 7.1470 (6.08) −1.5289 (7.27) 6.2360 (0.46) .5673 (1.13) .4203 (.061) −1.3571 (1.98) Other Quality .1477 (1.49) .1365 (1.58)   AF .9479 (1.58) .8144 (1.44) .7160 (0.93) .6977 (0.95) 2.0056 (2.38) 1.6533 (2.19) ARMY .5299 (0.78) .2793 (0.50) 1.0123 (0.84) .8918 (1.14) 1.6025 (1.46) 1.3328 (1.71) BMDO −.5610 (0.71) −.2555 (0.37) −.1657 (0.18) .0094 (0.01) −.1606 (0.14) .5494 (0.57) DARPA .2247 (0.34) .4794 (0.80) .5610 (0.75) .3082 (.045) 1.1246 (1.37) 1.0867 (1.44) DSWA .1123 (0.11) 1.4200 (1.67) −.7223 (0.46) .2664 (0.21) 1.5889 (0.97) 2.7578 (2.08) OSD .0204 (0.02) −.0455 (0.05)   Fast Track .2108 (0.43) .4113 (1.15) .6739 (0.77) .6526 (0.97) −.3007 (0.38) −.3426 (0.64) BMDO Match .8140 (1.29) .9040 (1.61) −.4197 (0.56) −.3575 (0.51) .8590 (0.95) .2153 (0.29) AGE −.2690 (1.73) −.2548 (2.06) .0979 (0.53) .0819 (0.55) −.0357 (0.19) −.0908 (0.60) SIZE .0017 (0.83) .0016 (0.98) .0053 (0.90) .0015 (0.42) .0065 (1.77) .0033 (1.21) Experience −.0110 (1.10) −.0047 (0.61) −.0318 (1.12) −.0114 (0.62) −.0286 (1.13) −.01239 (0.59) Ownership −.5125 (1.46) −.1524 (0.51) .2736 (0.57) .4000 (0.97) −.8089 (1.69) −.1729 (0.46) COMPUTER −.9787 (1.38)   −.6018 (0.61)   −.5348 (0.62)   ELECTRON −.5125 (0.89)   −.1314 (0.14)   .2177 (0.28)   MATERIAL −.7202 (0.85)   −.3861 (0.36)   −1.364 (1.03)   ENERGY −.6909 (0.83)   .0728 (0.07)   −.0787 (0.08)   ENVIRON −.0416 (0.05)   .4701 (0.36)   n 154 181 148 181 153 182 R2 .141 .105 .0867 .0609 .1012 .0559