National Academies Press: OpenBook

The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative (2000)

Chapter: Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach

« Previous: An Assessment of the Small Business Innovation Research Fast Track Program in the Southeastern States
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×

Evaluation of the Department of Defense Small Business Innovation Research Program and Fast Track Initiative: A Balanced Approach*

Robert B. Archibald and David H. Finifter

The College of William and Mary

EXECUTIVE SUMMARY

The purpose of this paper is to evaluate the Department of Defense (DoD) Small Business Innovation Research (SBIR) program and the Fast Track initiative. The SBIR program has grown and changed in its brief history, and the Fast Track Initiative is a recent innovation in the program. The SBIR program has a dual nature. It is designed to invigorate federal R&D efforts by expanding the participation of small business in the largest federal research programs and to increase the commercialization of innovative products and services that result from federal R&D efforts. Given the dual nature of the program it is important that evaluation of the program consider both the research and the commercial outcomes.

The SBIR Program—The SBIR Program was started in 1982 as a set-aside program for small business. It diverts a portion of the extramural research or research and development budgets of eleven federal agencies to fund the awards. The SBIR program has grown over time. Currently, each agency must devote not less than 2.5 percent of its extramural research and development budget to its SBIR program, representing a considerable increase from the original 0.2 percent of this budget. The SBIR awards are divided into three types. Phase I awards are small and intended to determine the scientific and technical merit and feasibility of the ideas. These awards are very competitive; overall there is an average of one award for every six to seven Phase I proposals. Phase II awards are designed to

*

Prepared for the May 5th SBIR Symposium sponsored by the Board on Science, Technology, and Economic Policy of the National Research Council, National Academy of Sciences (NAS).

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×

enable firms to do the research proposed in Phase I. Current Phase I awards normally have a maximum of $100,000 and Phase II awards have a maximum of $750,000, although these ceilings may be exceeded at the discretion of the program managers. Phase III awards do not actually involve SBIR funds. The terminology is used to describe situations in which actual production for agency use is funded through the continuation of a successful SBIR project.

The rationale for aid to small businesses generally relies on claims of discrimination in capital markets. The discrimination in capital markets is frequently called statistical discrimination. The fact that statistics indicate small businesses typically have higher default rates than large businesses is used to deny the loan to a small business applicant. Statistical discrimination leads to social inefficiency because small firms have less access to capital than they would if information were perfect.

When it created the SBIR program, Congress was also concerned with discrimination in government procurement. The concern here is with research productivity, i.e., a budget that relies heavily on large business results in lower productivity for research and development than would a budget that increases small business participation. Much of the mission-essential research carried out by government agencies is in areas that are highly innovative. In the industries in which small firms may have an advantage, it is sensible technology policy for the government to target some of its R&D funds on small firms.

The Fast Track Initiative—The 1996 Fast Track Initiative of DoD represents a continuation of the shift in emphasis in the SBIR award process toward commercial success. Under Fast Track firms with Phase I contracts which can interest outside investors in committing funds to further the development of the project increase their chances of obtaining Phase II funding and are eligible for bridge funding between Phase I and Phase II. The increase in the importance of commercial success is clear. A firm that does a piece of research that increases knowledge of a government laboratory provides a useful service to the government by aiding the ongoing research of the laboratory. Such a firm will, however, not be able to attract outside investors, so it will not be eligible for Fast Track. Fast Track is reserved for firms that are likely to be commercial successes through producing a product or service that can be directly sold, or whose product or service holds sufficient commercial promise that outside investors are willing to invest in its further development.

Fast Track fits a particular model of small business. It is designed for a small business that has technical expertise and a desire to use that technical expertise to develop a product or service that it can sell, either in the commercial marketplace or as a government contractor, or both. This firm has no particular desire to be a small business. There are two other types of firms that participate in the SBIR program. First, some successful small businesses have no desire to be big busi nesses. These firms are not growth-oriented. Some of these were created started

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×

by refugees from laboratories in large businesses and others by researchers who got their start in university or government laboratories. The founders of these firms were frustrated by the bureaucracy in the large business or some part of their academic or government job and became convinced to start or join a small business. Second, there are growth-oriented firms with products that are not yet commercially viable. These firms would like to expand and become big businesses, but the product or service they are developing is too far from its final state to be interesting to outside investors.

Survey of Technical Monitors—Our independent contribution to the evaluation is measurement of the quality and usefulness to the federal government of the research conducted by the SBIR firm, i.e., we provide measures of research success. The individuals who are in the best position to assess the quality and usefulness of SBIR research to the government are government scientists and engineers who monitor the research progress. Our evaluation takes the form of a survey of the technical monitors of the SBIR Phase II contracts. The technical monitors, called either Contracting Officer’s Technical Representative (COTR) or Technical Points of Contact (TPOC), are the DoD scientists or engineers who served as the liaison between the small business and the agency that awarded the contract.

One innovative aspect of the survey is that it was conducted via e-mail to make it practical and encourage a high response rate. Because it was designed for e-mail distribution, the survey was as short as possible while capturing key questions. We had a very high response rate for e-mails that were received by a COTR, although we were unable to find good e-mail addresses for a large number of COTRs. The database contains 379 SBIR Phase II contracts. The sample was drawn systematically. It includes all of the 1996 Fast Track projects and all of the BMDO Matching projects from 1992 to 1996 as well as a matched sample of regular Phase II projects. As such it does not represent a random sample of the entire DoD SBIR program. We have responses covering 195 (51.5%) of the contracts in the full sample. Our overall response rate for successful contacts, 78.9%, is very high.

The survey had two parts and covered five major areas. The first part of the survey focused on individual SBIR projects, measuring (1) research quality, (2) usefulness of the research, and (3) mission benefits of the research. The second part of the survey focused on the SBIR program, measuring (4) overall quality of SBIR proposals, and (5) impressions of Fast Track.

  1. Quality of Research—We asked two questions about the quality of research. One question asked for a rating of the quality of the research for the SBIR Phase II and the second asked for a rating of the quality of other research. For our sample, the mean value for the difference between the rating given to the research quality of the SBIR project and the rating given to other research is .025 with a standard deviation of 2.366.

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
  1. Usefulness of the Research—To discover whether or not the SBIR research was useful we ask survey respondents to indicate whether or not the research has affected the way that research is conducted in their unit/office. 25.94% indicated “No, the knowledge generated by this SBIR contract has had no impact on the other research we conduct or sponsor. The other 74.06% chose one of the responses that started with “Yes, this project produced results that have been useful to us, . . .”

  2. Mission Benefits—We asked the respondents to compare the mission benefits per dollar on the SBIR project with the average benefits per dollar on other research. 31.58% indicated that the SBIR project yielded more benefits per dollar, 42.11% indicated it yielded the same benefits per dollar, and 26.54% indicated it yielded less benefits per dollar.

  3. Overall Quality of Proposals—We asked each COTR about whether he or she was satisfied with the quality of the SBIR proposals his or her office received. 63.91% indicated that they had more good proposals than they could fund, 25.18% indicated that they had the same number of good proposals as they could fund, and 10.91% indicated that they had fewer good proposals than they could fund.

  4. Fast Track—We asked those COTRs who had been involved with a Fast Track about its effectiveness. 66.67% indicated that it was more effective than the normal program. 21.57% indicated that it was the same as the normal program, and 11.76% indicated that it was less effective than the normal program.

In summary, the DoD SBIR program received a very favorable evaluation from the technical monitors. The quality of the research in the SBIR program is indistinguishable from the quality of other research. Regarding usefulness of the research, over 74% of the responses indicated that the research was useful. Just over 73% of the responses indicated that a dollar spent on the SBIR project yielded the same or more benefits per dollar than other research efforts. Over 81% of the respondents said that they had the same, or more, good SBIR proposals than they could fund. And the Fast Track Initiative was rated as more effective than the normal SBIR program by two thirds of the respondents.

In the analysis of the survey results we divided the sample among the Fast Track projects, the BMDO Matching projects and the comparison group projects. Perhaps surprisingly, the Fast Track projects did very well in these comparisons. Despite the fact that the Fast Tract Initiative was designed to enhance the commercial success rate of its projects, according to our results the research outcomes of the Fast Track projects compared favorably to those in the other two samples.

Balanced Evaluation—The fundamental assumption of our approach to evaluation is that the two goals of the SBIR program should be given equal weights. Under this approach, projects that produce commercial success should count as successful SBIR projects, and projects that produce useful research for

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×

the contracting agency should count as successful SBIR projects. A proper evaluation of the program has to consider these two possible avenues to success for each project. This approach corrects a fundamental flaw in current evaluations, the fact that, exclusively, these evaluations have been separate evaluations of either commercial success or research success.

To accomplish a balanced evaluation we need information on both research outcomes and commercial outcomes. To obtain such data, we combine the results from our survey of COTRs with the results from a survey of firms conducted by Peter Cahill of BRTRC. The survey of firms was designed primarily to provide information on commercial outcomes. The linked survey responses therefore allow us to measure interactions between the research and commercial goals of the SBIR program. A majority of the projects for which we have information from technical monitors is contained in the data set of firm responses. There are 124 observations in the linked data set.

For given definitions of research success and commercial success, we can classify projects into one of four outcomes. Table 1 illustrates our evaluation strategy.

Table 1 is the basis for our evaluation of the SBIR program. There are several measures of research and commercial outcomes, and for some measures it is not obvious where one should draw the line between success and failure. For this reason, there are many ways one could classify projects. In the text we provide several examples using different definitions of the two types of success. Our major finding is that in almost all of the examples we considered, commercial and research success were statistically independent. This means we did not find a trade off between commercial success and research success. Also, we found that projects in the Fast Track Initiative generally did very well both in measures of research outcomes and measures of commercial outcomes.

Summary and Conclusions—There are three major conclusions in this paper.

  • The balanced evaluation we outline is the appropriate way to evaluate the SBIR program. Almost all of the previous evaluations have only considered commercial success. The quality of the research in the SBIR program has received scant attention. This paper provides the first evaluation that looks simultaneously at the two goals of the program.

TABLE 1 Classifications of SBIR Projects

 

Research Failure

Research Success

Commercial Failure

Group N—Neither a research nor a commercial success

Group R—Research successes that are not commercial successes

Commercial Success

Group C—Commercial successes that are not research successes

Group B—Both a commercial success and a research success

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
  • Our survey of technical monitors indicates that the DoD SBIR produces high quality research that in useful to the overall research effort of the department. Also, this high quality research does not appear to conflict with the SBIR goal of increasing the commercialization of the results of government R&D.

  • Contrary to our initial expectations, projects in the DoD Fast Track Initiative appear to produce research that is as good if not better than the research in the normal DoD SBIR program.

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
INTRODUCTION

The purpose of this paper is to evaluate the Department of Defense (DoD) Small Business Innovation Research (SBIR) program and the Fast Track Initiative. The SBIR program has grown and changed in its brief history, and the Fast Track Initiative is a recent innovation in the program. The results of surveys of technical monitors and SBIR awardees, which we report here in detail, indicate that these technical monitors have a strongly positive assessment of the Fast Track Initiative and the Fast Track firms have generally been very successful. The focus of this paper is on the appropriate evaluation of the SBIR program and the Fast Track Initiative.

The SBIR program has a dual nature. It is designed to invigorate federal research and development (R&D) efforts by expanding the participation of small business in the largest federal research programs and to increase the commercialization of innovative products and services that result from federal R&D efforts. Given the dual nature of the program, it is important that evaluation of the program consider both the research and the commercial outcomes.

Seven additional sections follow. In the second section, we give a brief description of the SBIR program and discuss its history. In the third section, we provide an outline of the theoretical rationale for the program. We introduce the Fast Track Initiative in the fourth section and explain its role in the context of the SBIR program. In the fifth section, we present evidence from a survey of technical monitors of SBIR contracts. In the sixth section, we provide an analysis of the results of the survey of technical monitors, focusing on research outcomes for the DoD SBIR program and the Fast Track Initiative. In the seventh section, we combine the survey of technical monitors with a survey of firms. The combined survey allows us to do a balanced evaluation of the DoD SBIR program and the Fast Track Initiative, including information on both goals of the SBIR program. The final section contains a summary and our conclusions.

DESCRIPTION OF THE SBIR PROGRAM

The SBIR program was started in 1982 as a set-aside program for small business. It diverts a portion of the extramural research or R& D budgets of 11 federal agencies to fund the awards. The SBIR program has grown over time. Currently, each agency must devote at least 2.5 percent of its extramural research R&D budget to its SBIR program, representing a considerable increase from the original 0.2 percent of this budget.

The SBIR awards are divided into three types. Phase I awards are small and intended to determine the scientific and technical merit and feasibility of the ideas. These awards are very competitive; overall there is an average of one award for every six to seven Phase I proposals. Phase II awards are designed to enable firms to do the research proposed in Phase I. They are granted to roughly one-third of the winners of Phase I awards. Phase III awards do not actually involve

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×

SBIR funds, but this terminology is used to describe situations in which actual production for agency use is funded through the continuation of a successful SBIR project. On occasion, private commercial funding also is described as Phase III funding. Current Phase I awards normally have a maximum of $100,000 and Phase II awards have a maximum $750,000, although these ceilings may be exceeded at the discretion of the program managers.

Each agency establishes its own policies and priorities regarding the categories of projects funded by its SBIR program, receives and evaluates proposals, selects awardees for SBIR funds, and makes the appropriate payments. The Small Business Administration (SBA) sets the schedule for the solicitation of proposals and, along with the Office of Science and Technology Policy, receives annual reports from each agency that runs an SBIR program.

To evaluate the SBIR program, it is important to identify the legislative intent for and goals of the program. The Small Business Innovation Development Act of 1982 was the result of a recommendation of the first White House Conference on Small Business in January 1980. The delegates to this conference voted to recommend the bill that authorized the creation of the SBIR program. There were several reasons this piece of legislation found support in the conference and eventually in the Congress. First, evidence suggested that small businesses had been having difficulty obtaining funds in general and had a declining share of federal R&D contracts. Second, several well-publicized studies indicated that small businesses were a very important source of job growth1, and the recessions of the early 1980s created a supportive climate for any proposal that could claim job creation potential. Third, a successful SBIR pilot project established by the National Science Foundation demonstrated that the program was feasible.

From the start, Congress had two goals for the SBIR program. A report of the Senate Committee on Small Business makes this clear.

The purpose of the bill is twofold: to more effectively meet R&D needs brought on by the utilization of small innovative firms (which have been consistently shown to be the most prolific sources of new technologies) and to attract private capital to commercialize the results of the Federal research. (U.S. Congress, 1981, p. 1)

The two primary goals for the program are also clear from the legislation. The 1982 act that created the SBIR program listed the following objectives of the program:

  1. to stimulate technological innovation;

  2. to use small business to meet Federal research and development needs;

  3. to foster and encourage participation by minority and disadvantaged persons in technological innovation; and

  4. to increase private sector commercialization of innovations derived from federal research and development. (96 Stat. 217)

1  

See, for example, David Birch (1981).

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×

The act was originally scheduled to expire on October 1, 1988, but during fiscal year 1986, Congress enacted legislation extending the law through September 1993 (99 Stat. 443). In 1992 the SBIR program was reauthorized. The Small Business Innovation Reauthorization Act of 1992 both raised the percentage of research expenditures dedicated to the SBIR program and increased the importance of the goal of commercializing SBIR projects. The goal of commercialization moved from being listed fourth in 1982 to second in 1992. This change in the ordering of the goals was purposeful and was reflected in important ways in the language describing the selection process after 1992. Specifically, the original language describing Phase I was:

  1. a first phase for determining, insofar as possible, the scientific and technical merit and feasibility of ideas submitted pursuant to SBIR program solicitations: (96 Stat. 218)

This language was amended as follows (the added language is underlined):

  1. a first phase for determining, insofar as possible, the scientific and technical merit and feasibility of ideas that appear to have commercial potential as described in subparagraph (B)(ii). submitted pursuant to SBIR program solicitations: (106 Stat. 4250)

For Phase II the change is more dramatic. The original language was:

  1. a second phase to further develop the proposed ideas to meet the particular program needs, the awarding of which shall take into consideration the scientific and technical merit and feasibility evidenced by the first phase and where two or more proposals are evaluated as being of approximately equal scientific and technical merit and feasibility, special consideration shall be given to those proposals that have demonstrated third phase, nonFederal capital commitments; (96 Stat. 218)

This was changed to:

  1. a second phase, to further develop proposals which meet particular program needs, in which awards shall be made based on the scientific and technical merit and feasibility of the proposals as evidenced by the first phase considering, among other things, the proposal ’s commercial potential, as evidenced by:

    1. the small business concern’s record of successfully commercializing SBIR or other research;

    2. the existence of second phase funding commitments from private sector or non-SBIR funding sources;

    3. the existence of third phase, follow-on commitments for the subject of the research; and

    4. the presence of other indicators of the commercial potential of the idea. (106 Stat. 4251)

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×

These are clear mandates to change the selection processes by increasing the importance of commercial potential. Under the 1982 legislation, ties between projects deemed to be of equal scientific and technical merit could be broken in favor of projects that were more likely to be commercially successful. The likelihood of commercialization was clearly a secondary concern. This was changed with the 1992 legislation, which placed commercialization on an equal footing with scientific and technical merit.

The 1996 Fast Track Initiative of DoD represents a continuation of the shift in emphasis in the SBIR award process toward commercial success. Under Fast Track, firms with Phase I contracts that can interest outside investors in committing funds to further the development of the project increase their chances of obtaining Phase II funding and are eligible for bridge funding between Phase I and Phase II. The increase in the importance of commercial success is clear. A firm that does a piece of research that increases the knowledge base of a government laboratory provides a useful service to the government by aiding the ongoing research of the laboratory. However, such a firm will not be able to attract outside investors, and so it will not be eligible for Fast Track. Fast Track is reserved for firms that are likely to be commercial successes through production of a product or service that can be directly sold, or whose product or service holds sufficient commercial promise that outside investors are willing to invest in its further development.

RATIONALE FOR THE SBIR PROGRAM

An important feature of the SBIR program is that it is a set-aside. It does not result from new monies appropriated by Congress. It results from Congress mandating that agencies engaged in research target a portion of their existing funds for research projects carried out by small businesses. Because of this, the SBIR program represents a redirection of R&D spending, not an expansion. R&D provides a public good and, for that reason, it is a sensible public expenditure.2 The question raised by the existence of the SBIR program is: Why does the government want to increase the involvement of small business in this activity?

We think of the SBIR program as addressing two failures: a market failure and a government procurement failure. First, the SBIR program is one of several programs designed to help small businesses. The loan and loan guarantee programs administered by the SBA provide other examples. These programs address a failure in credit markets. Second, the SBIR program seeks to correct deficiencies in federal procurement practices that lead to an excessive reliance on large businesses for federal R&D. Evaluations of the SBIR program should focus on how well the program corrects these two failures.

2  

See Griliches (1992) for a review of the literature on this point.

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
The SBIR Program as Aid to Small Businesses

The rationale for aid to small businesses generally relies on claims of discrimination in capital markets. Frequently called statistical discrimination,3 it occurs when, for example, a lender decides to make a loan to a large business, not to a small business, simply on the basis of size; that is, the two businesses are otherwise equally creditworthy. The fact that statistics indicate that small businesses typically have higher default rates than large businesses is used to deny the loan to a small business applicant.4 This kind of decision is cost-effective for the lender, but it is easy to see why government remedies are appealing. Statistical discrimination leads to social inefficiency because small firms have less access to capital than they would if information were perfect. This rationale for support of small business is generally recognized. One important question the SBIR program raises is: Is there anything about R& D activity or high-technology firms that exacerbates these problems?

The answer to this last question is probably “yes.” R&D is a much riskier investment than are many other types of investments that a firm might undertake. Because of this riskiness, R&D investment requires a much higher rate of return in capital markets than financing an expansion in current capacity. As Bronwyn Hall points out “asymmetric information between firms and investors implies that, to fund projects about which they do not have full information, investors will demand a ‘lemons’ premium in the form of a higher rate of return.” (1993, p. 290). This lemon effect for R&D investment applies to all firms, but it is very likely to be an even bigger problem for small firms. Josh Lerner (1999) explains how the SBIR program helps small firms to overcome these difficulties. He argues that a small firm that is successful in the competition for SBIR funds sends a signal of its capability to outside investors. In essence, to outside investors this certification indicates that winners of SBIR awards are less likely to produce R&D lemons.

One could argue that the special problems that small businesses have obtaining R&D funds might have been overcome by creating a special category of SBA loans or loan guarantees to support R&D. If the argument put forth by Lerner is correct, the current SBIR program is probably a superior alternative. The program gives small firms a chance to enter a competition in which federal technical experts judge their R&D proposals. A small firm’s successful competition in this arena signals that it is likely to be better than other small businesses in the commercial arena as well.

3  

See Arrow (1973) for an early discussion of statistical discrimination.

4  

Stiglitz and Weiss (1981) demonstrate why financial institutions’ response to this problem is to deny credit rather than simply to raise the interest rates they charge on loans to risky borrowers.

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
The SBIR Program as Aid to Federal Research Efforts

When it created the SBIR program, Congress was concerned with discrimination in government procurement. The 1981 Senate Report indicates that encouraging greater small business participation in federally sponsored R&D was an important part of the argument for the legislation:

Numerous studies have shown that small businesses are our Nation’s most efficient and fertile source of innovations. Yet only 3.5 to 4 percent of the Federal R&D dollar is spent with small firms. This underutilization of small business in Federal R&D programs is especially regrettable when considering the highly successful track record of small firms in generating jobs, tax revenue, and other economic and social benefits. (U.S. Congress, 1981, pp. 4-5.)

Such language clearly implies that the federal R&D procurement process had unwisely led to choices that limited the participation of small businesses.

The argument here is based on claims about research productivity; that is, a budget that relies heavily on large business results in lower productivity for R&D than would a budget that increases small business participation. Academic views about the relationship between firm size and innovation in the private sector have varied over time, but by the time the SBIR legislation was being considered, there was growing support for the notion that small firms might produce better research than large firms.5 Clearly, there are disadvantages of large size. For one, it may be more difficult for a large firm to take on a risky project than for a small firm. In a large firm, many layers of control have to approve a particular project, and this can discourage innovation. There is considerable evidence that frustration with unimaginative management leads to the creation of small high-technology firms by talented refugees from the laboratories of large businesses.

As researchers started to do empirical work on the relationship between firm size and innovation, support for the views that large scale was necessary for innovation diminished.6 Writing in 1970, F. M. Scherer summarized some of the evidence as follows:

The weight of the available quantitative evidence favors a conclusion that among the largest 500 or so U.S. industrial corporations, increases in size do not as a rule contribute positively to the intensification of R & D inputs or inventive outputs, and in more cases than not, giant scale has a slight to moderate stultifying effect, (p. 361)

With the advantage of additional time, Chris Freeman and Luc Soete (1997) report a similar finding in the summary to their chapter on the relationship between firm size and innovation:

5  

As an example, a study by the Organization for Economic Cooperation and Development concluded that “Small firms have shown remarkable ability as purveyors of innovations, in particular in industries characterized by high growth rates and technical change” (OECD, 1982, p. 5).

6  

See, for example, Rubenstein (1958), Hamberg (1963), and Roberts (1968).

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×

But when all is said and done, there is still impressive support for the view that small firm innovations have genuinely increased their share of the total in the final quarter of the twentieth century. (p. 239)

Zoltan Acs and David Audretsch (1990) add some important detail in their careful study using data for innovations. They find that innovative activity is hindered in concentrated markets. They also find that “the innovative advantage of small firms is promoted in industries that are highly innovative and that utilize a high component of skilled labor.” (p. 147)

Although not all of the arguments about R&D in the private sector necessarily carry over to the public sector, this last finding is important for the SBIR program. Much of the mission-essential research carried out by government agencies is in areas that are highly innovative. In the industries in which small firms may have an advantage, it is sensible technology policy for the government to target some of its R&D funds on small firms.

If it is the case that small firms are at least as likely to produce good R&D as are large firms, one might wonder why a government agency would spend its R&D budget predominantly at large firms. In some cases, government R&D efforts naturally by their very scale and complexity require large business; for example, few would want to fly a jet aircraft designed and engineered by a coalition of small businesses). Also, the reliance on large businesses might result from the same type of risk minimization behavior characterized by models of statistical discrimination. The perceived high variance of outcomes from small business might discourage procurement officials. Still, there are many types of government research that could be carried out by small business just as efficiently as by large business, and Congress was convinced that the optimal share for small business was larger than 3.5 to 4 percent.

THE FAST TRACK INITIATIVE
Components of Fast Track

The Fast Track Initiative has three components: First, a firm qualifies for Fast Track if it can link with outside investors who are willing to partially fund the Phase II research. The attraction for outside investors is that the SBIR program will match their investment. The willingness of outside investors signals the SBIR program that the project is very likely to lead to a commercial success. Also, the funds from the outside investors add to the research budget of the firm, further benefiting the government research effort. Second, a Fast Track firm receives bridge funds between Phase I and Phase II. These bridge funds allow the firm to maintain its research effort and research staff until the Phase II contract begins. Third, Fast Track contracts, as its name implies, have a smaller time gap between Phase I and Phase II.

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Alternative Models of Small Businesses

Fast Track fits a particular model of small business. It is designed for a firm that has technical expertise and a desire to use that technical expertise to develop a product or service that it can sell, either in the commercial marketplace or as a government contractor, or both. This firm has no particular desire to be a small business. It views itself as on the way to becoming a big business, and it has developed its product or service to the point that it can convince outside investors to back the project. Such firms can be characterized as “growth-oriented with a commercially viable product.”

Viewing the entire SBIR program as a program for growth-oriented firms with commercially viable products is prevalent. First, SBIR program managers speak of a successful SBIR firm as “graduating” when the firm hires worker number 501 and is no longer classified as a small business. On the basis of some of their statements, these program managers have the objective of graduating as many small businesses as they can. Second, this view is reflected in the emphasis placed on commercial success in evaluations of the SBIR program. A major General Accounting Office (GAO) report in 1992 was titled Small Business Innovation Research Program Shows Success But Could Be Strengthened. The report came to that conclusion based upon research that looked only at rates of commercialization. Presumably, if commercialization rates had been higher, the last clause would have been removed from the title.7 Third, in the abstract of his paper evaluating the SBIR program, Josh Lerner (1999) explains that “This paper examines the largest U.S. public venture capital initiative, the Small Business Innovation Research (SBIR) program . . . .” Taking the view that the SBIR program represents “public venture capital” focuses entirely on the commercialization goal and implies that all SBIR firms are growth-oriented with commercially viable products.8

There are two other types of firms that participate in the SBIR program. First, some successful small businesses have no desire to be big businesses. These firms are not growth oriented. Some were started by refugees from laboratories of large businesses and others by researchers who got their start in university or government laboratories. The founders of these firms were frustrated by the bureaucracy in the large business or some part of their academic or government job and became convinced to start or join a small business.9 Second, there are growth-

7  

The SBIR program was studied closely in 1992 in association with the congressional debates on its renewal. The other major study conducted by the SBA (1992) also focused exclusively on commercialization rates.

8  

The notion of public venture capital is inappropriate for the SBIR program in several other respects. In the SBIR program the government does not gain an equity holding in the firm, nor does it become involved in the management of the firm.

9  

Scherer (1970) puts the case for flight from big business clearly: “A direct consequence of this problem, which has been noted time and again in case histories and treatises on research management, is a bias away from really imaginative innovations in laboratories of large firms. But more important, inability to get ideas approved by higher management drives many of the most creative individuals out of large corporation laboratories to go it alone in their own ventures. During the two decades following World War II thousands of research-based new enterprises were founded by frustrated fugitives from the laboratories of such U.S. giants as Sperry Rand, IBM, Western Electric, Hughes Aircraft, Raytheon, and many others.” (p. 354)

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×

oriented firms with products that are not yet commercially viable. These firms would like to expand and become big businesses, but the product or service they are developing is too far from its final state to be interesting to outside investors.10

The 1992 testimony of James A. Block, President of Creare Inc. of Hanover, New Hampshire (U.S. Congress, 1992) provides an example of a firm that is not interested in growing. In discussions, DoD-SBIR program managers assured us that this is one of several cases we could have presented. Mr. Block outlined the commercialization philosophy of Creare Inc. as follows:

We remain an engineering service company that commercializes its product opportunities through licensing or the creation of separate product companies. Ten such companies trace their roots to Creare. We spin off products for commercialization not because we are blindly following an objective written over 30 years ago, but because we have found it very difficult to optimize both products and services in a single organization. (p. 358)

The point of this example is that Creare, no matter how many commercial successes it initiates, will always want to be a small engineering service firm.11

There is clearly hostility to firms such as Creare from some parts of the SBIR program. The April 17, 1998 GAO report evaluating the SBIR program indicates 10 current concerns about the SBIR program (GAO, 1998). Concern number 5 is of interest:

  1. the number of multiple-award recipients12 and the extent of their project-related activity after receiving SBIR funding.

Discussions about the SBIR program often focus on “SBIR mills,” that is, firms that are receiving a large number of SBIR awards and have no plans nor made

10  

We are implicitly assuming that each small business is only working with one technology. This is a simplification because it is clearly possible for a small business to be involved in many technologies. The analysis follows if we drop firm and replace it with “division ” or “technology group” or any other designation that indicates a portion of a firm’s effort focused on a technology that has yet to yield commercially viable products or services.

11  

While discussing the effects of trends toward networking and the growth in complexity of technology to discourage concentration in high-technology industries, Freeman and Soete (1997) suggest that Creare might not be an isolated example. They say, “These trends may be reinforced by the preferences of many engineers and scientists to work in smaller and more intimate organizations.” (p. 239)

12  

GAO defines multiple award recipients as Phase I awardees for that year who had received at least 15 Phase II awards in the preceding 5 years. There were 10 such firms in 1989 and 17 in 1996. The 17 companies in this category in 1996 had received between 15 and 61 SBIR Phase II awards in the preceding 5 years.

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×

any effort to commercialize the results of their research. Although “multiple-award recipients” is a less pejorative term than “SBIR mills,” the connotation is similar.

It is interesting to learn both why these firms are being singled out for special evaluation and how they are being evaluated. Commercialization, or the lack of it, is involved in the answer to both questions. The assumption of such an evaluation is that all small businesses are or should be growth oriented. If one views the SBIR program as a research and commercialization vehicle, then the goal is to carry out research, develop products, and eventually “graduate” small businesses, that is, help them to become big businesses. Under this interpretation, the existence of multiple-award recipients is a problem. These firms have received many awards but have not graduated. However, if one takes seriously the goal of the SBIR program to reduce the barriers to small firms as providers of federal R&D, and one believes that smaller may be better in some types of R&D, then the existence of multiple-award recipients is not a matter of concern. The model that relies on the graduation of firms ignores the fact that some commercialization takes place, as the Creare example illustrates, by firms spinning off new firms that concentrate on commercial development.

Attacks on multiple-award recipients take one of three approaches. First, some completely ignore the research component of the SBIR program. Second, some have the false impression that every small business would like to be a big business. Third, some argue that the multiple-award recipients are just good at writing proposals, but not good at producing research. This is possible, but only for a short time. There is enough continuity in the SBIR program for such a firm to be found out. Either the multiple-award recipients both write good proposals and do good research, or there is a serious problem with the integrity of those involved in the selection and monitoring of SBIR contracts. The intent of Congress is clear. As noted earlier, a firm’s past commercial success, or lack thereof, should be taken into account in the selection of Phase II projects. Our survey results and our interviews with program managers indicate that this dimension of the program is fully understood.

As a final comment on multiple-award recipients, it is interesting that there is no concern over such firms among big businesses. There are many big businesses that engage in a large amount of government R&D. They are perhaps best known as long-term successful defense contractors. They are not termed “mills,” and they get no special attention in GAO reports.

The third firm type, growth-oriented firms with not yet commercially viable products, represent riskier SBIR projects. In these firms, research is not sufficiently advanced to be able to demonstrate commercial potential to outside investors. The existence of these firms in the SBIR program raises a research policy question. Is the SBIR program incurring enough risk? If we view the SBIR program as a program entirely for growth-oriented firms with commercially viable products, the risk of research and commercial failures will be minimized.

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×

TABLE 1 Types of Small Businesses

Types of Small Businesses

Desire to Grow

Desire to Become Producers

Reliance on Government Funding

Appeal to Outside Investors

Suitability for Fast Track

Growth-oriented commercially viable product

High

High

Low Short Term

High Short Term

High

Non-growth-oriented

Zero

Low

High Long Term

Low Long Term

Low

Growth-oriented not yet commercially viable product

High

Moderate or High

High Short Term

Low Short Term

Low

However, a research program that focuses too much on reducing the risks will not push the technological frontiers as far and as fast as one that is more willing to take risks.13 The optimal amount of risk to include in a research program is difficult to determine, and we cannot pretend to know the answer, but this still is an important factor to consider when evaluating any aspect of the SBIR program. It is clear that the ultimate objective of R &D conducted by DoD is to enhance national defense. Ultimately, R&D should get a better weapon in the hands of the soldier, a better ship for the sailor, or a better plane for the pilot. The development process for the better weapon, ship, or plane can be very long. Research on the final stages of the process will more quickly be able to point to concrete benefits. In a world in which it is important to collect success stories to justify a program, it is very tempting to concentrate resources on projects that are currently commercially viable. It would be politically naive for those running the SBIR program to ignore the need for commercial success. Therefore, it is a challenge for agencies to balance short-run and long-run goals. There are small businesses doing good research on emerging technologies that should have a chance in the SBIR program. As they develop expertise with their technologies, it is possible that they will develop commercially viable products. It would be very shortsighted to crowd these firms out of the SBIR program.

Table 1 provides a summary of our discussion of the three types of small businesses.

13  

SBIR technical monitors recognize this trade-off. One of the comments we received in our survey of technical monitors illustrates this: “I expect half of my SBIRs to be outright bombs at the end, and no more than 10-20 percent to pan out into practical technologies. In contrast . . . of this group probably has a 50-75 percent hit rate on practical products; however, his product are typically incremental improved versions, whereas mine are more likely to be novel.”

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Fast Track’s Niche

The Fast Track Initiative was started as a limited experiment. Up to 5 percent of the DoD SBIR contracts could be in the Fast Track program. As Table 1 indicates, not all small R&D firms would find the Fast Track Initiative useful. Firms will not be suited for Fast Track either because of the nature of the firm or the nature of the research projects. In either case, these firms are contributing to the research goal of the SBIR program. They are also contributing to the commercialization goals of the program indirectly, either through spinning off new companies or commercializing by licensing, or by further developing technologies that have the potential to be commercialized.

The number and proportions of firms of each particular type is unlikely to be stable over time. One aspect of a program innovation such as Fast Track is that it is very likely to expand its niche. Fast Track provides powerful incentives for an SBIR participant to push its technology toward commercial viability. Also, Fast Track provides incentives for a particular type of firm to participate in the SBIR program. A small R&D firm with a product that is close to commercial viability has extra incentives to search for military applications for the product. This could expand participation in the SBIR program. In both instances the number of growth-oriented firms with commercially viable products in the SBIR program, Fast Track’s niche, will grow. There are limits to the potential size of this niche. A balanced evaluation of the Fast Track Initiative, one that accounts for both commercial and research success, will tell us how successful the Fast Track Initiative has been in filling that niche.

As a final comment on the Fast Track Initiative, it is interesting that our interviews with DoD program managers uncovered several program innovations at the branch level that appear to be responses to Fast Track. Some of the features of Fast Track have been incorporated in the regular SBIR program. For example, one of the program managers sets some funds aside to use as bridge money between Phase I and Phase II. At the discretion of the program manager, these funds allow the research to continue in the time period between the end of Phase I and beginning of Phase II funding. Another program manager has initiated a system in which other components of the branch are asked to share with the SBIR office in the funding of some projects. This mimics the part of Fast Track that requires outside investors to commit funds for Phase II. In this way the program manager can see that other units in the same branch are very interested in the research. These two innovations show how, by its example rather than by its use, Fast Track has changed the SBIR program within DoD. Also, it shows that program managers have been given the freedom to innovate and that they have taken initiative.

SURVEY OF TECHNICAL MONITORS

The fundamental assumption of our approach to evaluation is that the two goals of the SBIR program should be given equal weight. Under this approach,

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×

projects that produce commercial success should count as successful SBIR projects, and projects that produce useful research for the contracting agency should count as successful SBIR projects. A proper evaluation of the program has to consider these two possible avenues to success for each project. This approach corrects a fundamental flaw in current evaluations, the fact that, exclusively, these evaluations have been separate evaluations of either commercial success or research success.

Our independent contribution to a balanced evaluation is measurement of the quality and usefulness to the federal government of the research conducted by the SBIR firm; that is, we provide measures of research success. In this section, we discuss the survey we conducted to measure the research outcomes of SBIR awards. In the next section, we describe the results that link this information with a separate survey of the commercial success of the firms awarded SBIR contracts for these projects.

Description of the Survey of Research Outcomes

The individuals who know about the quality and usefulness of SBIR research to the government are government scientists and engineers who monitor the research progress. Our evaluation takes the form of a survey of the technical monitors of the SBIR Phase II contracts. The technical monitors, called either Contracting Officer’s Technical Representative (COTR) or Technical Points of Contact, are the DoD scientists or engineers who served as the liaison between the small business and the agency that awarded the contract. For simplicity, we refer to these individuals as the COTRs. Quite frequently, the COTR was involved in determining the topics for the SBIR competition, and in most cases the COTR had worked with the firm during Phase I as well as Phase II.

One potential flaw in relying on a survey of the technical monitors is that they may have too narrow a view of the innovation process or the SBIR program. It is quite possible that a particular SBIR project might not be deemed very high quality or very useful by a scientist in one laboratory, but it might have very important applications in some other area. Because the COTR might be unaware of the eventual success of a particular SBIR Phase II, we interviewed the program managers, that is, the individuals in charge of the SBIR program for the various military branches and research agencies in DoD. These individuals provided important perspectives on our survey results.

One innovative aspect of the survey is that it was conducted via e-mail to make it practical and encourage a high response rate. Because it was designed for e-mail distribution, the survey was as short as possible while capturing key questions. One clear advantage of the e-mail survey over a regular mail survey is that it dramatically shortens the time lag involved. The first day of the survey, we had many responses. Our response rate reveals the good news and the bad news with an e-mail survey. The good news is that we had a very high response rate for e-mails

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×

TABLE 2 Response Rates

Military Branch/Agencya

Sample Size

Successful Contacts

Completed Surveys

Percentage of Total

Percentage of Contacts

Air Force

66

45

32

48.5

71.1

Army

44

37

34

77.3

91.9

BMDO

167

89

68

41.3

76.4

DARPA

49

41

30

61.2

73.1

DSWA

4

4

4

100

100

NAVY

42

21

18

42.8

85.7

OSD

8

7

5

62.5

71.4

Total

380

244

191

50.3

78.3

aDARPA = Defense Advanced Research Projects Agency, DSWA = Defense Special Weapons Agency, OSD = Office of the Secretary of Defense.

that were received by a COTR. The bad news is that we have been unable to find good e-mail addresses for a large number of COTRs.

The database contains 380 SBIR Phase II contracts. The sample was drawn systematically and it includes all of the 1996 Fast Track projects and all of the Ballistic Missile Defense Office (BMDO) matching projects from 1992 to 1996 as well as a matched sample of regular Phase II projects. As such, it does not represent a random sample of the entire DoD SBIR program. Some of the analyses we present would be more appropriately performed with a random sample, and therefore the results should be interpreted with great care.

We have responses covering 191 (50.3%) of the contracts in the full sample. Table 2 accounts for our responses. Clearly, our biggest problem is making contact; our overall response rate for successful contacts, 78.3 %, is very high.14 The low response rates for BMDO and Navy result primarily from difficulties caused by retirements and missing e-mail information.

The survey had two parts and covered five major areas. The first part of the survey focused on individual SBIR projects, measuring (1) research quality, (2) usefulness of the research, and (3) mission benefits of the research. The second part focused on the SBIR program, measuring (4) overall quality of SBIR proposals, and (5) impressions of Fast Track. Our 191 responses represent responses to the first part of the survey. In the cases in which a COTR was assigned to more than one project in the sample, we only asked the questions in the second part of the survey once. There are 132 responses to the second part of the survey. The question about Fast Track was asked only of those COTRs who had experiences with Fast Track. We have 51 responses to that question.

14  

We defined a successful contact as an e-mail message that was not returned to us by the mailing system.

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Part 1: Responses about SBIR Phase II Projects

Area 1: Research Quality

One area of interest for any research program is the quality of the research. We asked two questions about the quality of research:

  1. On a 1 to 10 scale, where 10 represents the best research ever produced in your research unit/office or for your research unit/office and 1 represents the worst research ever produced in your research unit/office or for your research unit/office, rate the quality of the research in this particular SBIR contract.

Responses (n = 189): Mean = 7.069, Std. Dev. = 1.865.

  1. On the same 10-point scale, rate the average quality of the research projects conducted for your research unit/office from contracts other than SBIR contracts for the last three years.

Responses (n = 181): Mean = 7.064, Std. Dev. = 1.754.

We included Question 4 to provide a context for the responses to Question 3. Some people may be easy graders, giving high marks to any research project, while others are hard graders, giving low grades to any research project. The most interesting number is the difference between the responses to Question 3 and Question 4.

Previous studies contradict one another on the quality of SBIR research. As part of the assessment required by Congress, the GAO in 1989 studied the quality of the research in the SBIR program. The GAO report on the scientific quality of the SBIR program gave the program very high marks. The basic findings were based on a survey of 530 project officers who monitor SBIR research. The conclusion was:

  • Overall, respondents assessed 29 percent of the SBIR projects as being higher quality than non-SBIR research and indicated that about half of the SBIR projects were similar in overall quality to other research.

This suggests that the overall quality of SBIR research is roughly comparable to the quality of other federal research. A 1996 article by Jeffrey Mervis in Science indicates that there are those who dispute this claim. Using data on evaluation scores for research proposals at the National Institutes of Health, opponents of the SBIR program have shown that funded SBIR awards have lower scores than do other research projects.

For our sample, the mean value for the difference between the rating given to the research quality of the SBIR project and the rating given to other research is 0.025 with a standard deviation of 2.366 (n = 180). Clearly, there is no statistically significant difference between these ratings. We conclude that the quality of SBIR research is roughly equal to that of other contract research in the DoD.

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×

Area 2: Usefulness of the Research

To discover whether the SBIR research was useful, we asked survey respondents to indicate whether the research has affected the way that research is conducted in their unit/office.

  1. Has the research conducted for this SBIR contract affected the way that your research unit/office conducts research or the type of research your research unit/office obtains in other contracts? List as many as apply,

    1. No, the knowledge generated by this SBIR contract has had no impact on the other research we conduct or sponsor.

    2. Yes, this project produced results that have been useful to us, and we have tried to follow up on the ideas initiated in this SBIR contract by encouraging the firm to apply for additional SBIR awards.

    3. Yes, this project produced results that have been useful to us, and we have tried to follow up on the ideas initiated in this SBIR contract in other research we conduct or sponsor.

    4. Yes, but this project found a blind alley, so we have not followed up on this line of inquiry.

Responses (n = 183)15: (a) = 25.96%, (b) = 19.13%, (c) = 49.18%, (d) = 5.74%.

The responses to Questions 5 are quite positive for the SBIR program. Combining the two unambiguously positive responses, (b) and (c), we find that 68.30% of the SBIR had results encouraging enough to suggest further research. The high percentage for the (c) response indicates that the research that the SBIR program sponsors frequently affects the general DoD research program. The response of (a) to Question 5 appears to indicate a research failure, that is, the research project generated no follow-up. It is interesting that we found a few (d) responses, projects that produce useful research by finding a blind alley. This indicates that the SBIR program funds some risky projects.16

Area 3: Mission Benefits

The next question in the survey focused on mission benefits:

  1. In comparison to a dollar spent in your research unit/office on other R&D projects, did a dollar spent on this SBIR project:

    1. yield more benefits for your agency’s mission than the average dollar spent on other contracts sponsored by your research unit/office,

    2. yield the same level of benefits for your agency’s mission as the average dollar spent on other contracts sponsored by your research unit/office

    3. yield fewer benefits for your agency’s mission than the average dollar spent on other contracts sponsored by your research unit/office

15  

We have distributed the joint responses, for example, (b) and (c), evenly so that the total responses = 100%.

16  

It is important to recognize that all of our projects are Phase II projects. The prevalence of blind alleys would be much higher for Phase I projects.

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×

Responses (n = 186): (a) = 31.99%, (b) = 41.67%, (c) = 26.34%.

The responses to Question 6 also offer positive evidence for the SBIR program. We recognize that respondents were asked to make a difficult comparison, and some respondents pointed this out to us. They were asked to compare the benefits to the agency’s mission per dollar from the SBIR project with the average benefits to the mission per dollar of other sponsored research. This is a complicated comparison. Still, the results are consistent with findings on other dimensions of research output. A vast majority of the SBIR projects, 73.66%, were rated as beneficial or more beneficial per dollar than the average of other R&D projects. This is a strong endorsement of the research conducted in the SBIR program.

Part 2: Responses by COTRs About SBIR and Fast Track

Area 4: Overall Quality of Proposals

We asked each COTR about the whether he or she was satisfied with the quality of the SBIR proposals his or her office had received.

  1. In general, do you find that your research unit/office has had more (about the same, too few) good SBIR proposals than you can fund?

Responses (n = 131): more = 64.12%, same = 24.81%, too few = 11.07%.

These results reflect well on the DoD SBIR program. A large majority of the COTRs think that they have more good proposals than they can fund, indicating a very competitive program. The competitive nature of the SBIR program could indicate that the program is underfunded.

Area 5: Fast Track

We asked those COTRs who had been involved with a Fast Track to respond to the following question:

  1. If you have experience with the Fast Track program, is it more effective, less effective, or not different from the normal SBIR program in advancing the research program of your unit?

Responses (n = 51): more = 66.67%, less = 11.76%, not different = 21.57%.

This result suggests that Fast Track has a good reputation among those technical monitors who have been involved with it. It is unfortunate that the sample size is so low for this question. There were only 53 respondents who had been involved in any of the Fast Track SBIR contracts, and 2 of them failed to respond to Question 9.

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
FURTHER ANALYSIS OF THE TECHNICAL MONITOR SURVEY RESPONSES

In this section we evaluate the DoD SBIR program and the Fast Track Initiative with a focus on measures of quality, usefulness, and mission benefits derived from our survey of technical monitors.

COTR “Ownership”

When we constructed our survey of the technical monitors, we were concerned that the responses from COTRs who had a stake in the evaluation of the project might be biased in favor of showing favorable results. For this reason, we asked both whether or not the COTR was involved in defining or generating the topic for the Phase II project and when the COTR became involved with the SBIR firm. COTRs who were involved in defining or generating the topic as well as having been involved with the firm from before the Phase I proposal might be suspected of being biased in favor of the project. They might have taken some ownership of the research. Our interviews with program managers indicate that they thought this was possible.

Table 3 gives the results of responses from COTRs who might have experienced an ownership effect and the remaining COTRs in the areas of Research Quality, Usefulness of Research, and Mission Benefits. There is little evidence of an effect. In fact, although the difference is not statistically significant, the average quality rating is lower for the projects for which the COTR might have exhibited an ownership effect. The COTRs in the ownership group do seem to rate the projects as more useful. This result is statistically significant.17 Perhaps it not a surprise to see that the COTRs who had the most chance to shape the research found it most useful. The ratings of mission benefits indicate little difference at all. We interpret these results as saying that there is no bias in the ratings of research quality or mission benefits from an ownership effect.

Fast Track and BMDO Matching

The sample used in this study was designed to study Fast Track. It contained three types of SBIR projects: (1) Fast Track, (2) BMDO Matching contracts, and (3) a comparison group. The BMDO Matching sample was included because the BMDO SBIR program requires matching funds for Phase II as does Fast Track. BMDO Matching and Fast Track are not identical, but they do share the important feature that they require outside funds for participation in Phase II. Table 4 gives the results for these three samples.

17  

A chi-square test yields a statistic of 8.66, which exceeds the critical value of 7.815 for the 5% level.

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×

TABLE 3 Analysis of Ownership Effect

Area of Interest

Ownership Group

Remaining COTRs

Research Quality (Q3-Q4)

−.1279

.1373

Usefulness of Research (Q5)

a. no, not useful

17.9%

31.9%

b. yes, more SBIR

24.3%

15.2%

c. yes, general follow-up

55.1%

44.8%

d. yes, but blind alley

2.5%

8.1%

Mission Benefits (Q6)

a. more than average

32.5%

31.6%

b. same as average

44.8%

40.0%

c. less than average

23.8%

28.3%

It is interesting to compare these results to those from our direct question about Fast Track. Two-thirds of the 51 COTRs who had direct experience with Fast Track rated it as “more effective than the normal SBIR program.” Though none of the differences is statistically significant, generally the results in Table 4 are consistent with this assessment. The average Research Quality of Fast Track projects is highest and the percentage of “no, not useful” in the rating of the Usefulness of Research is lowest for the Fast Track projects.18 The only assessment in which Fast Track projects are not rated most highly is for projects rated as having “more than average” Mission Benefits per dollar, though the Fast Track research does have the highest percentage of “more” or “the same” level of benefits. Our conclusion is that these findings corroborate the opinion of the COTRs who had experience with Fast Track projects.

The results in Table 3 and Table 4 are for differences in means. It is possible that a closer look at the data, one that controls for other factors, will provide different results.

For each project, we have information on the following firm and program characteristics:

  1. number of prior Phase II contracts for the firm;

  2. whether the firm was woman or minority owned;

  3. number of employees of the firm;

  4. year of the award;

  5. major topic area of the project:

    1. computers,

    2. electronics,

18  

One of the program managers we interviewed indicated that the Fast Track projects should be expected to have better results because they had higher budgets. These projects have large additions to their research budgets from a source or sources outside the SBIR program.

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×

TABLE 4 Analysis of the Fast Track, BMDO Matching, and Comparison Group Samples

Area of Interest

Fast Track

BMDO Matching

Comparison Group

Research Quality (Q3-Q4)

n = 30

n = 51

n = 99

 

.4333

.1764

−.1767

Usefulness of Research (Q5)

n = 32

n = 52

n = 99

a. no, not useful

15.6%

39.4%

22.2%

b. yes, more SBIR

21.9%

18.3%

18.7%

c. yes, general follow-up

59.4%

38.5%

51.5%

d. yes, but blind alley

3.1%

3.9%

7.5%

Mission Benefits (Q6)

n = 32

n = 53

n = 101

a. more than average

28.1%

26.4%

36.16%

b. same as average

46.9%

48.1%

36.6%

c. less than average

25.0%

25.5%

27.2%

  1. materials,

  2. mechanical performance of vehicles, weapons, facilities,

  3. energy conservation and use,

  4. environment and natural resources,

  5. life sciences;

  1. branch that awarded the contract;

  2. whether the COTR might exhibit an ownership effect.

In Appendix A, we provide information on the three measures of research success by the program characteristics. Although there are some clear differences in the table in Appendix A, few of them are statistically significant.

Our research strategy is, holding other things constant, to search for relationships between these variables and the measures of quality, usefulness, and mission benefits. What type of results do we expect from this investigation? Consider, for example, the relationship between research quality and the number of prior Phase II contracts for the firm. If, holding other things constant, we found a statistically reliable positive relationship between these two measures, it would indicate that firms with longer track records in the SBIR program, on average, produced better research. One could infer from that finding that the overall quality of the SBIR program could be improved if such firms received a larger share of the contracts. On the other hand, if we found a statistically reliable negative relationship, the conclusion would be just the opposite. The overall quality of the SBIR program could be improved if such firms received a smaller share of the contracts. Either of these results would suggest that the DoD SBIR program needs to change its project selection to improve the overall quality of the research. If, however, there is not a statistically significant relationship, the overall

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×

quality of the projects cannot be improved by changing the project selection. In this case, the absence of statistically significant relationships indicates a well-run program.

Appendix B gives the results of our correlation and regression analyses. The correlation analysis indicates that there are few strong correlations between variables. The three measures of research success—(1) research quality, (2) whether the research was useful to the further research effort, and (3) whether the contract yielded more than average benefits to the mission—are related to one another. Each of these correlations is above .2. The correlations between these indicators of research success and the firm and program characteristic variables are, however, not high. None of these correlations is as high as .2, and most of them are below .1. Also, there is little evidence of correlation among the program and firm characteristics.

The regression analyses for quality indicated only one statistically significant relationship. The year of the project, with more recent projects being rated as being higher quality, was statistically significant. The lack of statistically reliable findings is, as we explained earlier, an indication that the DoD SBIR program is efficiently managed, at least in terms of the characteristics we could measure. There do not appear to be statistically reliable relationships that could be exploited to raise the average quality of the research.

The results for the logit regressions of the indicator variable that measured whether the research was useful to the further research effort (based on the responses to Question 5) indicate completely statistically insignificant results. Interestingly, the effect we found in the bivariate analysis between the ownership and the ratings of usefulness of the research disappears in the multivariate analysis. Again, as with research quality, the statistically insignificant findings suggest that the SBIR program cannot exploit existing information to enhance the usefulness of the projects.

The final regression, for the indicator variable for projects that provided more than average mission benefits (based on responses to Question 6), contains some statistically significant results. The coefficients for the indicator variables for the Air Force, the Army, and DSWA are all positive and statistically significant (the excluded variable for this group of indicator variables was BMDO). Also, the coefficient for projects in the Electronics topic category was positive and statistically significant (the omitted variable for this group of indicator variables was Materials).

In sum, these results provide a positive evaluation of the DoD SBIR program. Measures of research quality, the usefulness of the research, and the benefits to the agency mission are generally not statistically significantly related to information available at the time of the award of the contract. This indicates that there are no easy improvements in research performance.

These results do suggest a way of thinking about some issues surrounding the SBIR program. For example, the lack of a relationship between the measures

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×

of research success and the number of prior Phase II contracts provides no support for restrictions on multiple-award recipients. The results also provide no particular support for multiple-award recipients; that is, there is no evidence that they receive more SBIR contracts because they do better-than-average research. According to our results, the research performance of these firms is indistinguishable from that of firms with less experience in the program. Finally, these results suggest that Fast Track contracts and BMDO Matching contracts are similar to other SBIR contracts in terms of their research output. There is no evidence that the research outputs of the SBIR program have been compromised by the emphasis on commercial success from the Fast Track Initiative.

COMBINING MEASURES OF RESEARCH AND COMMERCIAL SUCCESS

In this section, we combine the results from our survey of COTRs with the results from a survey of firms conducted by Peter Cahill of BRTRC. The survey of firms was designed primarily to provide information on commercial outcomes. The linked survey responses therefore allow us to measure interactions between the research and commercial goals of the SBIR program. A majority of the projects for which we have information from technical monitors are contained in the data set of firm responses. There are 122 observations in the linked data set.

Table 5 gives a summary of the findings of the two surveys and the characteristics of the linked data set. We considered three measures of commercial success: (1) having actual sales, (2) having expected sales, and (3) having either actual or expected sales. The data indicate that a vast majority of the firms (or their licensees) have or expect to have sales from a product or service developed through SBIR. Generally, the linked data set is representative of the two larger data sets. Firms that have already had sales are slightly underrepresented in the linked data set. Interestingly, the linked sample includes projects that the technical monitors rated as slightly better for all three of the measures of research success than in the full COTR data, but the differences are very small.

Our analytical strategy is similar to the one we used for the analysis of the COTR survey. The major difference is that success now has two dimensions. Those involved in the SBIR program in many agencies have long struggled with defining success in the program. In our view, there is no single measure of success. Success in the SBIR program is inherently two-dimensional. For given definitions of commercial success and research success, we can place projects into four groups:

  1. Group N—neither a research success nor a commercial success,

  2. Group R—research successes that are not commercial successes,

  3. Group C—commercial successes that are not research successes,

  4. Group B—both a research success and a commercial success.

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×

TABLE 5 Results of Both Surveys

Variable

Survey of COTRs (%)

Survey of Firms (%)

Linked Survey (%)

Firm survey questions

     

8. Firm or licensee has had sales of products, process, services or other sales from the technology

 

42.6

36.9

11. Firm or licensee expected to have sales in the next three years but has yet to have sales

 

50.4

54.9

8. & 11. Firm or licensee has had or expects to have sales

 

93.0

91.87

COTR survey questions

     

3 & 4. Quality of the SBIR research is greater than the average quality of non-SBIR research

46.6

 

48.4

5. The SBIR research has been useful to the government agency

74.0

 

74.5

6. The SBIR project yielded more benefits per dollar than average

32.0

 

33.9

These groups are then the focus for analysis.

We have several potential measures of research and commercial outcomes and, for some measures, it is not obvious where one should draw the line between success and failure. For this reason, there are many ways one could classify the projects. We provide two examples.

Example 1: Sales and Usefulness of Research

In this example we use Question 8 on the survey of firms:

  1. Has your company and/or licensee realized any actual sales of products, process, services or other sales from the technology developed during this project?

For research success we will use Question 5 from the COTR survey:

  1. Has the research conducted for this SBIR contract affected the way that your research unit/office conducts research or the type of research your research unit/office obtains in other contracts?

Combining these two definitions of success yields the results in Table 6.

Given these classifications, there are two interesting questions. First, are commercial and research success related? For these data, they are not. The chisquare statistic for the test of independence is 1.525, which is not sufficiently high to reject independence. Second, what are the correlates of assignment into

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×

TABLE 6 Classification of Research and Commercial Success Using Usefulness of Research and Sales

 

Research Success

Commercial Success

Yes (74.5%)

No (24.5%)

Yes (37.9%)

Group B, 30 (25.8%)

Group C, 14 (12.1%)

No (62.1%)

Group R, 56.5 (48.7%)

Group N, 15.5 (13.4%)

the various classifications? Table 7 presents information on this question. The table shows the prevalence of projects with various characteristics in the linked sample and for the projects in the two extreme categories: Group B, both a research and a commercial success, and Group N, neither a research nor a commercial success. The entries in the table are percentages of projects in the particular classification that are part of a particular group. For example, 25.4% of the full sample are Fast Track projects, but 19.4% of the Category B projects are Fast Track projects.

Though none of the differences in proportions are statistically significant, the results for Fast Track are mixed. Fast Track projects are slightly underrepresented in Category B, which does not reflect well on Fast Track, and very underrepresented in Category N, which reflects well on Fast Track. If it is more important to avoid Category N than it is to hit Category B, as seems sensible, then these are essentially positive results for Fast Track. The results for BMDO Matching projects appear to be uniformly inferior. Overall the Comparison group firms seem better in this comparison. Firms with more than three prior Phase II awards appear to be inferior by these measures also. In contrast, having fewer than three prior Phase II awards results in a statistically significant lower proportion of firms in Category N. This is the only statistically significant difference for this table. Finally, there does not appear to be a clear pattern for results based upon the number of employees in the firm.

TABLE 7 Analysis of Example 1 Classifications by Groups

Variable

Full Sample (%)

Group B (%)

Group N (%)

Fast Track

25.4

19.4

6.7

BMDO Matching

25.4

16.1

53.3

Comparison

49.2

64.5

40.0

Prior Phase II ≤ 3

81.1

90.3

60.0

Prior Phase II > 3

18.9

9.7

40.0

Employees ≤ 10

58.0

54.8

50.0

Employees > 10

42.0

45.2

50.0

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Example 2: Sales and Mission Benefits

In this example we keep the same definition of commercial success and change the definition of research success. We base research success on Question 6 in the COTR survey:

  1. In comparison to a dollar spent in your research unit/office on other R&D projects, did a dollar spent on this SBIR project yield more/the same/less benefits for your agency’s mission than the average dollar spent on other contracts sponsored by your research unit/office.

We define research success as yielding more benefits than the average on other contracts, a very strict definition of research success.

In this case, the test of the hypothesis of independence yields a chi-square statistic of 1.209, which is not statistically significant.

Table 9 presents the analysis of these categories by the groups we used above. In these comparisons, again Fast Track is underrepresented in Group B, but this time Fast Track is also overrepresented in Group N as well, the uniformly poor rating. The BMDO Matching sample has mixed results, overrepresented in Group B and slightly underrepresented in Group N. None of the results for Fast Track or the BMDO Matching is statistically significant. Prior Phase II experience is again not related to favorable outcomes according to these findings. Again, firms with three or fewer prior Phase II awards were significantly less likely statistically to be in Category N. The results for firm size indicate that small firms are more than proportionately related to being both a commercial and a research success. For firm size, the result that firms with 10 or fewer employees were more likely to be in Group B is statistically significant.

The two examples illustrate that this type of analysis depends critically on the way one defines success. In Table 6, we defined research success in terms of whether the research in the SBIR provided results that were useful to the agency—a minimal notion of success. In Table 8, the research success was reserved for projects that provided more benefits to the agency mission than the average contract—a much stricter notion of research success. In Table 8, with the lower hurdle for research success, over 86% of the topics registered some type of success. In Table 8, with the higher hurdle for research success, just over 56% of the projects registered some type of success.

TABLE 8 Classification of Research and Commercial Success Using Mission Benefits and Sales

 

Research Success

Commercial Success

Yes (33.9.5%)

No (66.1%)

Yes (38.1%)

Group B, 18 (15.3%)

Group C, 14 (22.9%)

No (61.9%)

Group R, 22 (18.6%)

Group N, 51 (43.2%)

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×

TABLE 9 Analysis of Example 2 Classifications by Groups

Variable

Full Sample (%)

Group B (%)

Group N (%)

Fast Track

25.4

18.2

31.4

BMDO Matching

25.4

31.8

24.5

Comparison

49.2

50.0

55.9

Prior Phase II ≤ 3

81.1

95.4

72.5

Prior Phase II > 3

18.9

4.6

27.5

Employees ≤ 10

58.0

77.3

52.1

Employees > 10

42.0

22.7

47.9

SUMMARY AND CONCLUSIONS

Our summary and conclusions come under four headings: (1) conclusions about appropriate evaluation of the SBIR program, (2) results for the general DoD SBIR program, (3) results for the Fast Track Initiative, and (4) suggestions for future research.

Appropriate Evaluation of the SBIR Program

The SBIR program is difficult to evaluate. Most previous evaluations have focused on commercial success and have been based on surveys of firms. The quality of the research in the SBIR program has received little attention. This paper provides the first evaluation that looks simultaneously at the two goals of the program. There are three methodological points we want to emphasize.

  1. The SBIR program has two goals: to increase the participation of small business in federal R&D and to increase the commercialization of innovations developed as a result of federal R&D. An appropriate evaluation has to account for both of the program goals.

  2. E-mail is a good medium to evaluate outcomes of the research. The technical monitors—the DoD scientists and engineers who worked with the firms—were generally quite willing to respond to an e-mail survey.

  3. Our analysis of the responses from the survey of technical monitors and the survey of firms illustrates the appropriate methodology for evaluating a program with two clear goals. One objective of any agency ’s SBIR program should be to minimize the projects that are neither a research nor a commercial success. Our methodology illustrates the first attempt to evaluate progress for this objective.

Results Regarding General DoD SBIR Program

Several of our results provide an evaluation of the performance of the overall DoD SBIR program as a research program. In general, the results are very positive. There are five findings we would like to highlight:

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
  1. Based on our survey responses, the quality of the research conducted in the SBIR program is indistinguishable from other research conducted by or sponsored by DoD.

  2. Based on our survey responses, a large majority (74.04 percent) of the SBIR projects were rated as having produced results that were useful for the ongoing research effort at DoD.

  3. Based on our survey responses, a large majority (73.66 percent) of the SBIR projects were rated as having produced the same or more mission benefits per dollar than other R&D projects of DoD.

  4. Based on our survey, a substantial majority (61.14 percent) of the technical monitors indicated that they had more good SBIR proposals than they could fund, and only a small proportion (11.07 percent) indicated that they had fewer good SBIR proposals than they could fund.

  5. Using multivariate analysis, we found no clear evidence that the research success of the DoD SBIR program could be improved based upon information available to program managers prior to the award of the contract.

The SBIR program is being considered for reauthorization in the next Congress. One of the questions that will come up in that debate is whether it is sensible to expand the percentage of the R&D budget set aside for the SBIR program. This is a complex issue. Our results indicate that, at the current size, the quality of DoD SBIR research is high and there are more good proposals than can be funded. Although these results are not sufficient to suggest an expansion of the program, they probably represent necessary conditions.

Results Regarding the Fast Track Initiative

The sample we used was designed to study the Fast Track Initiative. We gathered information on Fast Track in two ways. We asked the technical monitors who had experience with the Fast Track Initiative for a direct evaluation and, in the analysis of our results, we separated the sample to see if performance by Fast Track projects differed from performance by the comparison groups. The Fast Track Initiative generally is viewed very favorably and the results for Fast Track projects are impressive.

  • Two-thirds of the technical monitors who had experience with a Fast Track project rated Fast Track as more effective in advancing the research program in their unit. Only 11.76 percent of these respondents rated Fast Track as less effective in advancing the research program in their unit.

  • Though the differences were not statistically significant, the quality rating for Fast Track projects indicated an advantage for Fast Track compared to other projects.

  • Though the differences were not statistically significant, the percentage of Fast Track projects rated as being useful for the research effort was higher than for non-Fast Track projects.

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×

These results are impressive because the Fast Track Initiative was designed to improve the commercial success, not the research success, of the DoD SBIR program. Given the current size of Fast Track—5 percent of the DoD SBIR program—there clearly is not a trade-off between commercial and research success. Although perhaps unexpected, the lack of any reduction in research success may result from the fact that Fast Track projects have larger research budgets. One of the clear advantages of the Fast Track Initiative is that it attracts funds to the federal R&D effort. Our results suggest this has been beneficial for research.

Issues Deserving Further Research

One limitation of the research reported in this paper is the sample design. A correct evaluation of the DoD SBIR program requires a random sample. The linked sample provides a very interesting view of outcomes in the SBIR program. The objective of program managers should be to reduce the number of failures, that is, projects that are neither research nor commercial successes. It would be very important to search for the correlates of projects that are classified in this category from a random sample of SBIR projects. The preliminary analysis with our sample uncovered some interesting findings. In particular, it appears that firms with three or fewer prior Phase II contracts are underrepresented in the failures whereas firms with more than three prior Phase II contracts are overrepresented. It would be very interesting to see if such findings hold for a random sample.

ACKNOWLEDGMENTS

We would like to thank Joe Heyman, Thayer Sheets, and Bob Yang of NASA-Langley Research Center for valuable discussions about the workings of the SBIR program, Charles Wessner and John Horrigan (NAS) for useful comments on an earlier version of this paper, Peter Cahill (BRTRC) for help with data, and public policy graduate students Stephen Bowman, Jennifer Jebo, Randy Rosso, and Keith Wandtke for valuable research assistance. The above-listed individuals are not responsible for any shortcomings of the paper. The views expressed are those of the authors and do not represent the view of the College of William and Mary. Please do not cite without permission.

REFERENCES

Acs, Zoltan J., and David B. Audretsch. 1990. Innovation and Small Firms. Cambridge, MA: MIT Press.

Arrow, Kenneth. 1973. “The theory of discrimination,” pp. 3-31 in Discrimination in Labor Markets, Orley Ashenfelter and Albert Rees, eds. Princeton, NJ: Princeton University Press.

Birch, David. 1981. “Who creates jobs,” The Public Interest 65(Fall):3-14.

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×

Freeman, Chris, and Luc Soete. 1997. The Economics of Industrial Innovation. Cambridge, MA: MIT Press.

Griliches, Zvi. 1992. “The search for R&D spillovers,” Scandinavian Journal of Economics 94:S29-S47.

Hall, Bronwyn H. 1993. “Industrial research during the 1980s: Did the rate of return fall? ” Brookings Papers: Microeconomics 2:289-343.

Hamberg, Dan, 1963. “Invention in the industrial research laboratory,” Journal of Political Economy (April):95-115.

Lerner, Josh. 1999. “The government as venture capitalist: The long-run effects of the SBIR program,” Journal of Business. July, v. 72 3, pp. 285-297.

Mervis, Jeffrey D. 1996. “A $1 Billion ‘Tax’ on R&D Funds,” Science 272:942-944.

OECD (Organization for Economic Cooperation and Development. 1982. Innovation in Small and Medium Firms. Paris: OECD.

Roberts, Edward B. 1968. “Entrepreneurship and technology,” Research Management (July):249-266.

Rubenstein, A. H. 1958. Problems Financing New Research-Based Enterprises in New England. Boston MA: Federal Reserve Bank.

SBA (Small Business Administration). 1992. Results of Three-Year Commercialization Study of the SBIR program . Washington, D.C.: U.S. Government Printing Office.

Scherer, F. M. 1970. Industrial Market Structure and Economic Performance. New York: Rand McNally College Publishing.

Stiglitz, J. E., and A. Weiss. 1981. “Credit rationing in markets with incomplete information,” American Economic Review 71:393-409.

U.S. Congress, House. 1992. SBIR and Commercialization: Hearing Before the Subcommittee on Technology and Competitiveness of the House Committee on Science, Space, and Technology, on the Small Business Innovation Research [SBIR] Program (testimony of James A. Block, President of Creare Inc.) pages 356-361.

U.S. Congress, Senate Committee on Small Business. 1981. Small Business Research Act of 1981. S.R. 194, 97th Congress.

U.S. General Accounting Office. 1989. Federal Research: Assessment of the Small Business Innovation Research Program, GAO/RCED-89-39, Washington, D.C.: GAO.

U.S. General Accounting Office 1992. Small Business Innovation Research Program Shows Success But Can Be Strengthened, RCED-92-32. Washington, D.C.: GAO.

U. S. General Accounting Office. 1998. Federal Research: Observations on the Small Business Innovation Research Program, RCED-98-132. Washington, D.C.: GAO.

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×

APPENDIX A

Means of Variables

This table gives the means of the variables for various subsamples of the data. For simplicity these means are for the respondents who answered both Question 5 and Question 6. This causes some of the means to differ slightly from those reported in the text.

TABLE A1 COTR Survey Results

 

Useful

Mission Benefit

 

Quality

Q5 %

Q6 %

Subsample

N

Q3-Q4

b,c,d

a

b

Total

184

.0511

73

32

42

0 Prior Phase II

86

.2500

79

33

42

1-10 Prior

78

−.0395

69

32

40

>10 Prior

20

−.4000

65

25

50

Women/Min.

47

.2727

66

38

36

1-9 Employees

82

.4494

74

40

33

10-19 Emp

38

.0429

68

26

47

20-50 Emp.

33

−.6613

73

18

52

>50 Emp.

31

−.2419

77

29

48

Fast Track

31

.5862

84

29

48

92 Award

16

−.5625

63

19

56

93 Award

21

.5000

86

33

43

94 Award

30

−.5000

60

20

27

95 Award

33

.0156

73

48

30

96 Award

63

.2756

77

31

49

Computers

31

−.1207

71

26

42

Electronics

83

.3063

70

36

42

Materials

12

−1.6667

58

8

58

Mechanical

9

.6111

78

44

22

Energy Cons.

15

−.2000

73

27

47

Environment

4

.0000

100

50

50

Life Sciences

1

−4.000

100

0

0

Air Force

28

.5357

79

43

32

Army

31

.3214

87

39

32

BMDO

67

.0308

63

25

49

DARPA

33

−.2419

76

33

39

DSWA

4

.6250

75

75

0

Navy

18

−.6470

72

17

50

OSD

3

−.3333

100

0

100

Ownership

77

−.0205

82

32

44

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×

APPENDIX B

Multivariate Analyses

This appendix presents the correlation and regression results discussed in Section VI. The variable definitions are given below followed by the correlation matrix and the estimated regressions. Because there were several missing values for the topic categories, for each of the three dependent variables, SBIR Quality, Useful, and Benefits, there are two regressions, one using all the variables, and one excluding the variables for topic categories. The regressions for Useful and Benefits are logit regressions.

Variable

Definition

SBIR Quality

a rating on a scale of 1 to 10 for the quality of the SBIR project

Other Quality

a rating on the same scale of the average quality of non SBIR research conducted in the last three years

Useful

a categorical variable equal to one if the SBIR project’s research results were useful

Benefits

a categorical variable equal to one if the SBIR project’s benefits to the mission (per dollar) were greater than the benefits to the mission (per dollar) from other projects

AF

a categorical variable equal to one if the project was an Air Force funded project

ARMY

a categorical variable equal to one if the project was an Army funded project

BMDO

a categorical variable equal to one if the project was a BMDO funded project

DARPA

a categorical variable equal to one if the project was a DARPA funded project

DSWA

a categorical variable equal to one if the project was a DSWA funded project

NAVY

a categorical variable equal to one if the project was a Navy funded project

OSD

a categorical variable equal to one if the project was an OSD project

Fast Track

a categorical variable equal to one if the project was a Fast Track project

BMDO Match

a categorical variable equal to one if the project was a BMDO Matching project

AGE

age of the project computed as 97 minus the fiscal year for the project older projects will have larger values.

SIZE

size of the SBIR firm measured as the number of employees

Experience

experience in the SBIR program measured as the number of prior Phase II awards for the firm

Ownership

a categorical variable for those projects for which the COTR was involved with defining or generating the topic and was involved with the firm before the Phase I proposal

COMPUTER

a categorical variable equal to one if the topic area was Computers

ELECTRON

a categorical variable equal to one if the topic area was Electronics

MATERIAL

a categorical variable equal to one if the topic area was Materials

VEHICLES

a categorical variable equal to one if the topic area was Mechanical Performance of Vehicles, Weapons, or Facilities

ENERGY

a categorical variable equal to one if the topic area was Energy Conservation and Use

ENVIRON

a categorical variable equal to one if the topic area was Environment and Natural Resources

LIFE SCIENCES

a categorical variable equal to one if the topic area was Life Sciences

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×

TABLE B1 Correlation Matrix

 

(1)

(2)

(3)

(4)

(5)

(6)

(7)

(8)

(9)

(10)

(1) SBIR Quality

1.000

                 

(2) Other Quality

.1571

1.000

               

(3) Useful

.3840

.0095

1.000

             

(4) Benefits

.3731

−.1403

.2533

1.000

           

(5) AF

.0762

−.0503

.0494

.1254

1.000

         

(6) ARMY

.0639

.0487

.1487

.0606

−.1908

1.000

       

(7) BMDO

−.1048

−.1424

−.1763

−.1108

−.3270

−.3270

1.000

     

(8) DARPA

.0244

.1051

.0669

.0391

−.1986

−.1986

−.3403

1.000

   

(9) DSWA

.0815

.0375

.0141

.1408

−.0657

−.0657

−.1125

−.0683

1.000

 

(10) NAVY

−.0681

.0882

−.0296

−.1061

−.1452

−.1452

−.2488

−.1511

−.0500

1.000

(11) OSD

−.0116

−.0158

−.0397

−.1143

−.0736

−.0736

−.1262

−.0766

−.0253

−.0560

(12) Fast Track

.0967

−.0202

.1156

−.0043

−.1087

.3841

−.1700

.0013

−.0657

−.0445

(13) BMDO Match

−.0124

−.0754

−.1998

−.0730

−.2699

−.2699

.8353

−.2809

−.0929

−.2053

(14) AGE

−.2291

−.1500

−.0493

−.0561

−.0871

−.3673

.4206

−.0236

.0057

−.0080

(15) SIZE

.0362

.0593

−.0315

.0487

.0017

.0842

−.1507

−.1068

.0868

.1263

(16) Experience

−.0054

.0030

−.0592

−.0293

.1310

.0019

−.0791

−.0993

.0400

.0400

(17) Ownership

.0187

.1119

.1432

.0105

.1978

.1978

−.3723

−.0402

.0210

.1209

(18) COMPUTER

−.0202

.0908

−.0837

−.0336

.1134

.0739

−.3167

.0584

−.0697

.1847

(19) ELECTRON

.0284

−.1153

−.0313

.0835

−.0599

−.1508

.1832

−.0552

.0159

−.0021

(20) MATERIAL

−.1122

.0511

−.1031

−.1425

−.0632

−.1215

.2378

−.0697

−.0418

−.0209

(21) VEHICLES

.1016

.0353

.0355

.0638

.1772

−.0999

−.0123

.0309

−.0344

−.0760

(22) ENERGY

−.0752

−.0388

.0173

−.0312

−.0220

−.1313

.1092

.0761

−.0452

−.0329

(23) ENVRION

.0304

.0375

.0967

.0599

.0368

−.0657

−.1125

.1312

−.0226

.0756

(24) LIFE SCIENCES

−.1266

.0399

.0480

−.0505

.1706

−.0326

−.0339

−.0558

−.0112

−.0248

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×

(11)

(12)

(13)

(14)

(15)

(16)

(17)

(18)

(19)

(20)

(21)

(22)

(23)

(24)

                           
                           
                           
                           
                           
                           
                           
                           
                           

1.000

                         

.0183

1.000

                       

−.1041

−.2699

1.000

                     

−.1504

−.3897

.3039

1.000

                   

.1858

−.0589

−.1164

−.0223

1.000

                 

.1191

−.1469

−.0474

.0233

.5007

1.000

               

.0575

.1065

−.3380

−.1786

.1866

.0212

1.000

             

.0986

.1134

−.2539

−.1656

−.0237

−.0493

.0354

1.000

           

.0517

.1521

.1149

−.1332

−.0013

.0091

−.0876

−.4171

1.000

         

–.0469

−.1215

.2110

.3569

−.0063

.1089

−.0260

−.1289

−.2504

1.000

       

−.0386

−.0306

.0292

.0607

.0494

−.0138

.0062

−.1060

−.2059

−.0636

1.000

     

−.0507

−.0767

.0832

.1345

−.0977

−.0476

.0622

−.1393

−.2705

−.0836

−.0688

1.000

   

−.0253

−.0657

−.0929

.1176

−.0898

.0247

.0210

−.0697

−.1353

−.0418

−.0344

−.0452

1.000

 

−.0126

−.0326

−.0460

.0444

−.0342

−.0288

−.0649

−.0345

−.0671

−.0207

−.0170

−.0224

−.0112

1.000

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×

TABLE B2 Regression Equations

 

SBIR Quality

Useful

Benefits

Constant

7.1470

(6.08)

−1.5289

(7.27)

6.2360

(0.46)

.5673

(1.13)

.4203

(.061)

−1.3571

(1.98)

Other Quality

.1477

(1.49)

.1365

(1.58)

 

AF

.9479

(1.58)

.8144

(1.44)

.7160

(0.93)

.6977

(0.95)

2.0056

(2.38)

1.6533

(2.19)

ARMY

.5299

(0.78)

.2793

(0.50)

1.0123

(0.84)

.8918

(1.14)

1.6025

(1.46)

1.3328

(1.71)

BMDO

−.5610

(0.71)

−.2555

(0.37)

−.1657

(0.18)

.0094

(0.01)

−.1606

(0.14)

.5494

(0.57)

DARPA

.2247

(0.34)

.4794

(0.80)

.5610

(0.75)

.3082

(.045)

1.1246

(1.37)

1.0867

(1.44)

DSWA

.1123

(0.11)

1.4200

(1.67)

−.7223

(0.46)

.2664

(0.21)

1.5889

(0.97)

2.7578

(2.08)

OSD

.0204

(0.02)

−.0455

(0.05)

 

Fast Track

.2108

(0.43)

.4113

(1.15)

.6739

(0.77)

.6526

(0.97)

−.3007

(0.38)

−.3426

(0.64)

BMDO Match

.8140

(1.29)

.9040

(1.61)

−.4197

(0.56)

−.3575

(0.51)

.8590

(0.95)

.2153

(0.29)

AGE

−.2690

(1.73)

−.2548

(2.06)

.0979

(0.53)

.0819

(0.55)

−.0357

(0.19)

−.0908

(0.60)

SIZE

.0017

(0.83)

.0016

(0.98)

.0053

(0.90)

.0015

(0.42)

.0065

(1.77)

.0033

(1.21)

Experience

−.0110

(1.10)

−.0047

(0.61)

−.0318

(1.12)

−.0114

(0.62)

−.0286

(1.13)

−.01239

(0.59)

Ownership

−.5125

(1.46)

−.1524

(0.51)

.2736

(0.57)

.4000

(0.97)

−.8089

(1.69)

−.1729

(0.46)

COMPUTER

−.9787

(1.38)

 

−.6018

(0.61)

 

−.5348

(0.62)

 

ELECTRON

−.5125

(0.89)

 

−.1314

(0.14)

 

.2177

(0.28)

 

MATERIAL

−.7202

(0.85)

 

−.3861

(0.36)

 

−1.364

(1.03)

 

ENERGY

−.6909

(0.83)

 

.0728

(0.07)

 

−.0787

(0.08)

 

ENVIRON

−.0416

(0.05)

 

.4701

(0.36)

 

n

154

181

148

181

153

182

R2

.141

.105

.0867

.0609

.1012

.0559

Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 211
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 212
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 213
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 214
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 215
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 216
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 217
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 218
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 219
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 220
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 221
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 222
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 223
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 224
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 225
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 226
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 227
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 228
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 229
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 230
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 231
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 232
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 233
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 234
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 235
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 236
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 237
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 238
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 239
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 240
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 241
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 242
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 243
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 244
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 245
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 246
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 247
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 248
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 249
Suggested Citation:"Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach." National Research Council. 2000. The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative. Washington, DC: The National Academies Press. doi: 10.17226/9985.
×
Page 250
Next: Role of the Department of Defense in Building Biotech Expertise »
The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative Get This Book
×
Buy Hardback | $80.00 Buy Ebook | $64.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

In 1992, Congress for the first time explicitly directed the federal agencies making SBIR grants to use commercial potential as a criterion for granting SBIR awards. In response, the Department of Defense developed the SBIR Fast Track initiative, which provides expedited decision-making for SBIR awards to companies that have commitments from outside vendors. To verify the effectiveness of this initiative, the DoD asked the STEP Board to assess the operation of Fast Track. This volume of original field research includes case studies comparing Fast Track and non-Fast Track firms, a large survey of SBIR awardees, and statistical analyses of the impact of regular SBIR and Fast Track awards. Collectively, the commissioned papers and the findings and recommendations represent a significant contribution to our understanding of the SBIR program.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!