3
Scope and Methodology

3.1
STUDY SCOPE

This National Research Council (NRC) study focuses on the National Science Foundation’s (NSF’s) Small Business Innovation Research (SBIR) program, examining the program’s performance in meeting each of the following four SBIR legislated objectives:

  1. Stimulate technological innovation;

  2. Increase commercialization of innovation;

  3. Use small business to meet federal R&D needs;

  4. Foster participation by minority and disadvantaged persons in technological innovations.

The study also examines the effectiveness of the NSF’s management of its SBIR program, including resources, topic definition, solicitations, proposal selection, commercialization assistance, and general oversight of the program.

3.2
METHODOLOGY DESIGN

This study of the NSF’s SBIR program and the study’s findings and recommendations are based on a research methodology designed by the Committee, approved by an independent panel of experts, and described in more detail in the study’s “Methodology Report.”1 The study uses multiple methods to allow

1

National Research Council, An Assessment of the Small Business Innovation Research Program: Project Methodology, Washington, DC: The National Academies Press, 2004.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 41
3 Scope and Methodology 3.1 STuDY SCOPE This National Research Council (NRC) study focuses on the National Sci- ence Foundation’s (NSF’s) Small Business Innovation Research (SBIR) program, examining the program’s performance in meeting each of the following four SBIR legislated objectives: 1. Stimulate technological innovation; 2. Increase commercialization of innovation; 3. Use small business to meet federal R&D needs; 4. Foster participation by minority and disadvantaged persons in technologi- cal innovations. The study also examines the effectiveness of the NSF’s management of its SBIR program, including resources, topic definition, solicitations, proposal selec- tion, commercialization assistance, and general oversight of the program. 3.2 METHODOLOGY DESIGN This study of the NSF’s SBIR program and the study’s findings and recom- mendations are based on a research methodology designed by the Committee, approved by an independent panel of experts, and described in more detail in the study’s “Methodology Report.”1 The study uses multiple methods to allow 1National Research Council, An Assessment of the Small Business Innoation Research Program: Project Methodology, Washington, DC: The National Academies Press, 2004. 

OCR for page 41
 SBIR AT THE NATIONAL SCIENCE FOUNDATION cross-checking and confirmation of findings that emerge from any single method. Multiple methods are also necessary to examine the several objectives of the SBIR program, which vary in their relative tractability to quantitative and qualita- tive approaches. The core methodologies used in generating the data underlying the findings of this report are: surveys of firms that participated in the SBIR program; interviews with NSF SBIR officials and program managers; review of program documents and other relevant literature; analysis of program data; and case studies of firms. 3.3 METHODS uSED 3.3.1 Surveys Three surveys were sent to SBIR firms: (1) Phase II Survey; (2) Firm Sur- vey; and (3) Phase I Survey.2 The first listed survey was focused on a sample of Phase II NSF SBIR grants. This Phase II Survey had 162 respondents. The focus of the Phase II Survey was on specific grants and their commercial outcomes. The second survey explored the SBIR program’s influence on small busi- nesses more generally. Responding to this survey were 137 companies identified primarily as NSF Phase II grant recipients. The survey asked questions about the background experience of each firm’s founders, the receipt of Phase I and Phase II awards, and the influence of the SBIR program on firm founding, com- mercial progress, and growth. The third survey was focused on a sample of Phase I NSF SBIR grants, primarily for the purpose of identifying and learning more about those Phase I grants that did not receive a direct follow-on Phase II grant. The Phase I Survey had 248 respondents, of which 135 received Phase I grants that were not followed by a direct follow-on Phase II grant, and 113 received Phase I grants that were followed by a direct follow-on Phase II grant. Those that did receive a follow-on Phase II grant were asked if they received assistance and from whom. The methodologies of these three surveys are described in more detail in Appendixes B and C. These descriptions reveal the initial sample sizes, adjust- ments to the samples, response rates, and statistics on response rates. These appendixes also provide the survey instruments as well as survey results for indi- vidual questions. The number of respondents to each question and the base num- bers used to calculate percentage responses are also given. In considering these firm survey results, it is worth keeping in mind the possibility of response biases that could significantly affect the survey results. For example, it may be possible that some of the firms that could not be found have been unsuccessful and folded. 2For the NRC Phase II Survey, 35 percent of the overall sample of 457 responded. This represents 48 percent of the awards contacted. For the NRC Phase I Survey, the response rate was 10 percent of a sample size of 2,458. See Appendixes B and C.

OCR for page 41
 SCOPE AND METHODOLOGY It may also be possible that unsuccessful firms were less likely to respond to the survey. (See Appendixes B and C for details about the survey methodologies, survey instruments, and responses to individual questions.) NSF SBIR Program Manager Survey. A separate survey was sent to the relatively small group of NSF SBIR program managers. NSF SBIR officials declined having their program managers complete the survey and provided a single response to this survey prepared by the NSF SBIR program’s senior advisor, Dr. Joseph Hennessey. 3.3.2 Interviews with NSF SBIR Managers Multiple interviews were conducted with NSF SBIR program officials and program managers. The interviews explored various aspects of program opera- tions and how they have changed over time, past and current management prac- tices, and possible program modifications. 3.3.3 Review of Program Documents and Data Documents of the NSF SBIR program that were reviewed for the study included internal copies of evaluation studies, related data tables and success stories, survey and interview guides, success stories known as “nuggets,” annual reports, reports by the Committee of Visitors, staff presentations, program guide- lines, solicitation materials, examples of funded projects, and other materials. The NSF SBIR Office provided data on proposals and grants by individual records. Analysis of these data informed the program description and often figured in the analysis of program outputs. 3.3.4 Case Studies of Firms Case studies were prepared for ten selected companies located in seven states: California, Ohio, Tennessee, Minnesota, Arizona, Vermont, and Michigan. The companies are developing ten different technologies from a variety of dis- ciplines, including software, electrochemical processes, information technology for pest monitoring and control, electronics, manufacturing processes, and nanomaterials. The oldest company is more than 20 years old and the youngest was just 2 years old at the time of the interview. The smallest company has about a dozen employees, the largest nearly 150. Among the case-study companies are a university spin-off, a large company spin-off, a small company spin-off, a company started by a graduate student, a company started by university faculty, a company started by a retired large com- pany executive, several companies started by entrepreneurs leaving other com-

OCR for page 41
 SBIR AT THE NATIONAL SCIENCE FOUNDATION Box A Multiple Sources of Bias in Survey Response Large innovation surveys involve multiple sources of bias that can skew the results in both directions. Some common survey biases are noted below. • Successful and more recently funded firms are more likely to respond. Research by Link and Scott (2005) demonstrates that the probability of obtain- ing research project information by survey decreases for less recently funded projects and that it increases the greater the award amount.a Nearly 40 percent of respondents in the NRC Phase II Survey began Phase I efforts after 1998, partly because the number of Phase I awards increased, starting in the mid- 1990s, and partly because winners from more distant years are harder to reach. They are harder to reach as time goes on because small businesses regularly cease operations, are acquired, merge, or lose staff with knowledge of SBIR awards. • Success is self-reported. Self-reporting can be a source of bias, although the dimensions and direction of that bias are not necessarily clear. In any case, policy analysis has a long history of relying on self-reported performance measures to represent market-based performance measures. Participants in such retrospective analyses are believed to be able to consider a broader set of allocation options, thus making the evaluation more realistic than data based on third-party observation.b In short, company founders and/or principal investiga- tors are, in many cases, simply the best source of information available. • Survey sampled projects at firms with multiple awards. Projects from firms with multiple awards were underrepresented in the sample because they could not be expected to complete a questionnaire for each of dozens or even hundreds of awards. • Failed firms are difficult to contact. Survey experts point to an “asymmetry” in their ability to include failed firms for follow-up surveys in cases where the firms no longer exist.c It is worth noting that one cannot necessarily infer that the SBIR project failed; what is known is only that the firm no longer exists. • Not all successful projects are captured. For similar reasons, the NRC Phase II Survey could not include ongoing results from successful projects in firms that merged or were acquired before and/or after commercialization of the project’s technology. The survey also did not capture projects of firms that did not respond to the NRC invitation to participate in the assessment. • Some firms may not want to fully acknowledge SBIR contribution to project success. Some firms may be unwilling to acknowledge that they received impor- tant benefits from participating in public programs for a variety of reasons. For example, some may understandably attribute success exclusively to their own efforts. • Commercialization lag. While the NRC Phase II Survey broke new ground in data collection, the amount of sales made—and indeed, the number of projects that generate sales—are inevitably undercounted in a snapshot survey taken at a single point in time. Based on successive data sets collected from the National

OCR for page 41
 SCOPE AND METHODOLOGY 14 12 Sales (Millions of Dollars) 10 8 6 4 2 0 1 2 3 4 5 6 7 8 9 10 11 12 survey taken Years after Phase II Award FIGURE A-1 Survey Bias due to Commercialization Lag. Institutes of Health SBIR award recipients, it is estimated that total sales from all responding projects will likely be on the order of 50 percent greater than can be captured in a single survey.d This underscores the importance of follow-on research based on the now-established survey methodology. Figure B-A-1 a Albert N. Link and John T. Scott, Evaluating Public Research Institutions: The U.S. Ad- vanced Technology Program’s Intramural Research Initiative, London: Routledge, 2005. b While economic theory is formulated on what is called ‘revealed preferences,’—meaning individuals and firms reveal how they value scarce resources by how they allocate those resources within a market framework—quite often, expressed preferences are a better source of information, especially from an evaluation perspective. Strict adherence to a revealed preference paradigm could lead to misguided policy conclusions because the paradigm assumes that all policy choices are known and understood at the time that an individual or firm reveals its preferences and that all relevant markets for such preferences are operational. See Gregory G. Dess and Donald W. Beard, “Dimensions of Organizational Task Environments,” Administrative Science Quarterly 29 (1984):52–73, and Albert N. Link and John T. Scott, Public Accountability: Evaluating Technology- Based Institutions, Norwell, MA: Kluwer Academic Publishers, 1998. c Link and Scott, Evaluating Public Research Institutions: The U.S. Advanced Technology Program’s Intramural Research Initiative op. cit. d Data from the National Institutes of Health indicates that a survey taken two years later would reveal very substantial increases in both the percentage of firms reaching the market and the amount of sales per project. See National Research Council, An Assessment of the Small Busi- ness Innovation Research Program at the National Institutes of Health, Charles W. Wessner, ed., Washington, DC: The National Academies Press, forthcoming.

OCR for page 41
 SBIR AT THE NATIONAL SCIENCE FOUNDATION panies, and one firm started by a professor-husband and entrepreneur-wife team. The case studies include two woman-owned companies, one of which is actively operated by a woman who is also a member of a minority group. The sample includes companies that were able to commercialize a product very quickly and companies whose technologies are taking considerable time to commercialize. Commercialization strategies of these firms include licensing agreements, contract research, sales of product produced in-house, commercial- ization partnerships, as well as the sale of technology or the company itself to a larger company. The companies’ annual revenues range from $2 million to about $24 mil- lion. This includes companies whose share of annual revenue contributed by the SBIR program and other government grants comprise as little as 4 percent and as much as 70 percent. The ten companies selected for the case studies do not represent a random sample. The companies were selected to provide companies of different ages and size, pursuing different technologies, located in different parts of the country, with differing forms of ownership, and with some, although varying degrees of, commercial success. Some of the companies are university spin-offs; some are company spin-offs; some are neither. Some received many SBIR grants; some relatively few. Some continue to obtain a high percentage of their funding from government sources; others have reduced the percentage to low numbers. To illustrate different types of successful outcomes, six of the cases selected were identified by NSF as “stars.” In addition, all of these cases selected are revenue- earning SBIR-funded companies. The process for selecting the case studies is explained in detail in Appendix D.