Assessing program outcomes against these four objectives entails numerous methodological challenges. These challenges are discussed in detail in the National Research Council’s (NRC) Methodology Report.2
Assessment usually involves comparison—comparing programs and activities, in this case. Three kinds of comparison seem possible: with other programs at each agency, between SBIR programs at the various agencies, and with early-stage technology development funding in the private sector, such as venture capital activities. Yet, the utility of each of these three types of comparison is limited.
Other award programs at the agencies have fundamentally different objectives, such as promoting basic research through grant programs (at the National Institutes of Health [NIH] and the National Science Foundation [NSF]), developing capacity (awards for university infrastructure), or training. There are often no other dedicated programs for innovative small businesses. And no other programs for small businesses have as a primary goal the commercial exploitation of research. This fundamental difference in objectives makes it difficult to usefully compare an SBIR program with other programs at the relevant agency.
Comparisons between SBIR programs at different agencies appear superficially more useful, but must be regarded with considerable caution. As discussions in Chapter 1 of this volume and in the separate agency volumes indicate, the widely differing agency missions have shaped the agency SBIR programs, focusing them on different objectives and on different mechanisms and approaches. Agencies whose mission is to develop technologies for internal agency use via procurement—notably the Department of Defense (DoD) and the National Aeronautics and Space Administration (NASA)—have a quite different orientation from agencies that do not procure technology and are instead focused on developing technologies for use outside the agency.
Finally, SBIR might be compared with venture capital (VC) activities, but there are important differences. VC funding is typically supplied later in the development cycle when innovations are in, or close to, market—most venture investments are made with the expectation of an exit from the company within three years. VC investments are typically larger than SBIR awards—the average investment made by VC firms in a company was over $7 million in 2005,
National Research Council, An Assessment of the Small Business Innovation Research Program: Project Methodology, Charles W. Wessner, ed. Washington, DC: The National Academies Press, 2004, pp. 20-21. For a broader discussion of the scope and limitations of surveys by the University of Michigan Survey Research Center, see Robert M. Groves et al., Survey Methodology, Wiley-IEEE, 2004.