National Academies Press: OpenBook
« Previous: 2 Findings and Recommendations
Suggested Citation:"3 Scope and Methodology." National Research Council. 2008. An Assessment of the SBIR Program at the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/11929.
×

3
Scope and Methodology

3.1
STUDY SCOPE

This National Research Council (NRC) study focuses on the National Science Foundation’s (NSF’s) Small Business Innovation Research (SBIR) program, examining the program’s performance in meeting each of the following four SBIR legislated objectives:

  1. Stimulate technological innovation;

  2. Increase commercialization of innovation;

  3. Use small business to meet federal R&D needs;

  4. Foster participation by minority and disadvantaged persons in technological innovations.

The study also examines the effectiveness of the NSF’s management of its SBIR program, including resources, topic definition, solicitations, proposal selection, commercialization assistance, and general oversight of the program.

3.2
METHODOLOGY DESIGN

This study of the NSF’s SBIR program and the study’s findings and recommendations are based on a research methodology designed by the Committee, approved by an independent panel of experts, and described in more detail in the study’s “Methodology Report.”1 The study uses multiple methods to allow

1

National Research Council, An Assessment of the Small Business Innovation Research Program: Project Methodology, Washington, DC: The National Academies Press, 2004.

Suggested Citation:"3 Scope and Methodology." National Research Council. 2008. An Assessment of the SBIR Program at the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/11929.
×

cross-checking and confirmation of findings that emerge from any single method. Multiple methods are also necessary to examine the several objectives of the SBIR program, which vary in their relative tractability to quantitative and qualitative approaches. The core methodologies used in generating the data underlying the findings of this report are: surveys of firms that participated in the SBIR program; interviews with NSF SBIR officials and program managers; review of program documents and other relevant literature; analysis of program data; and case studies of firms.

3.3
METHODS USED

3.3.1
Surveys

Three surveys were sent to SBIR firms: (1) Phase II Survey; (2) Firm Survey; and (3) Phase I Survey.2

The first listed survey was focused on a sample of Phase II NSF SBIR grants. This Phase II Survey had 162 respondents. The focus of the Phase II Survey was on specific grants and their commercial outcomes.

The second survey explored the SBIR program’s influence on small businesses more generally. Responding to this survey were 137 companies identified primarily as NSF Phase II grant recipients. The survey asked questions about the background experience of each firm’s founders, the receipt of Phase I and Phase II awards, and the influence of the SBIR program on firm founding, commercial progress, and growth.

The third survey was focused on a sample of Phase I NSF SBIR grants, primarily for the purpose of identifying and learning more about those Phase I grants that did not receive a direct follow-on Phase II grant. The Phase I Survey had 248 respondents, of which 135 received Phase I grants that were not followed by a direct follow-on Phase II grant, and 113 received Phase I grants that were followed by a direct follow-on Phase II grant. Those that did receive a follow-on Phase II grant were asked if they received assistance and from whom.

The methodologies of these three surveys are described in more detail in Appendixes B and C. These descriptions reveal the initial sample sizes, adjustments to the samples, response rates, and statistics on response rates. These appendixes also provide the survey instruments as well as survey results for individual questions. The number of respondents to each question and the base numbers used to calculate percentage responses are also given. In considering these firm survey results, it is worth keeping in mind the possibility of response biases that could significantly affect the survey results. For example, it may be possible that some of the firms that could not be found have been unsuccessful and folded.

2

For the NRC Phase II Survey, 35 percent of the overall sample of 457 responded. This represents 48 percent of the awards contacted. For the NRC Phase I Survey, the response rate was 10 percent of a sample size of 2,458. See Appendixes B and C.

Suggested Citation:"3 Scope and Methodology." National Research Council. 2008. An Assessment of the SBIR Program at the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/11929.
×

It may also be possible that unsuccessful firms were less likely to respond to the survey. (See Appendixes B and C for details about the survey methodologies, survey instruments, and responses to individual questions.)

NSF SBIR Program Manager Survey. A separate survey was sent to the relatively small group of NSF SBIR program managers. NSF SBIR officials declined having their program managers complete the survey and provided a single response to this survey prepared by the NSF SBIR program’s senior advisor, Dr. Joseph Hennessey.

3.3.2
Interviews with NSF SBIR Managers

Multiple interviews were conducted with NSF SBIR program officials and program managers. The interviews explored various aspects of program operations and how they have changed over time, past and current management practices, and possible program modifications.

3.3.3
Review of Program Documents and Data

Documents of the NSF SBIR program that were reviewed for the study included internal copies of evaluation studies, related data tables and success stories, survey and interview guides, success stories known as “nuggets,” annual reports, reports by the Committee of Visitors, staff presentations, program guidelines, solicitation materials, examples of funded projects, and other materials.

The NSF SBIR Office provided data on proposals and grants by individual records. Analysis of these data informed the program description and often figured in the analysis of program outputs.

3.3.4
Case Studies of Firms

Case studies were prepared for ten selected companies located in seven states: California, Ohio, Tennessee, Minnesota, Arizona, Vermont, and Michigan. The companies are developing ten different technologies from a variety of disciplines, including software, electrochemical processes, information technology for pest monitoring and control, electronics, manufacturing processes, and nanomaterials.

The oldest company is more than 20 years old and the youngest was just 2 years old at the time of the interview. The smallest company has about a dozen employees, the largest nearly 150.

Among the case-study companies are a university spin-off, a large company spin-off, a small company spin-off, a company started by a graduate student, a company started by university faculty, a company started by a retired large company executive, several companies started by entrepreneurs leaving other com-

Suggested Citation:"3 Scope and Methodology." National Research Council. 2008. An Assessment of the SBIR Program at the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/11929.
×

Box A

Multiple Sources of Bias in Survey Response

Large innovation surveys involve multiple sources of bias that can skew the results in both directions. Some common survey biases are noted below.

  • Successful and more recently funded firms are more likely to respond. Research by Link and Scott (2005) demonstrates that the probability of obtaining research project information by survey decreases for less recently funded projects and that it increases the greater the award amount.a Nearly 40 percent of respondents in the NRC Phase II Survey began Phase I efforts after 1998, partly because the number of Phase I awards increased, starting in the mid-1990s, and partly because winners from more distant years are harder to reach. They are harder to reach as time goes on because small businesses regularly cease operations, are acquired, merge, or lose staff with knowledge of SBIR awards.

  • Success is self-reported. Self-reporting can be a source of bias, although the dimensions and direction of that bias are not necessarily clear. In any case, policy analysis has a long history of relying on self-reported performance measures to represent market-based performance measures. Participants in such retrospective analyses are believed to be able to consider a broader set of allocation options, thus making the evaluation more realistic than data based on third-party observation.b In short, company founders and/or principal investigators are, in many cases, simply the best source of information available.

  • Survey sampled projects at firms with multiple awards. Projects from firms with multiple awards were underrepresented in the sample because they could not be expected to complete a questionnaire for each of dozens or even hundreds of awards.

  • Failed firms are difficult to contact. Survey experts point to an “asymmetry” in their ability to include failed firms for follow-up surveys in cases where the firms no longer exist.c It is worth noting that one cannot necessarily infer that the SBIR project failed; what is known is only that the firm no longer exists.

  • Not all successful projects are captured. For similar reasons, the NRC Phase II Survey could not include ongoing results from successful projects in firms that merged or were acquired before and/or after commercialization of the project’s technology. The survey also did not capture projects of firms that did not respond to the NRC invitation to participate in the assessment.

  • Some firms may not want to fully acknowledge SBIR contribution to project success. Some firms may be unwilling to acknowledge that they received important benefits from participating in public programs for a variety of reasons. For example, some may understandably attribute success exclusively to their own efforts.

  • Commercialization lag. While the NRC Phase II Survey broke new ground in data collection, the amount of sales made—and indeed, the number of projects that generate sales—are inevitably undercounted in a snapshot survey taken at a single point in time. Based on successive data sets collected from the National

Suggested Citation:"3 Scope and Methodology." National Research Council. 2008. An Assessment of the SBIR Program at the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/11929.
×

FIGURE A-1 Survey Bias due to Commercialization Lag.

  • Institutes of Health SBIR award recipients, it is estimated that total sales from all responding projects will likely be on the order of 50 percent greater than can be captured in a single survey.d This underscores the importance of follow-on research based on the now-established survey methodology.

  

aAlbert N. Link and John T. Scott, Evaluating Public Research Institutions: The U.S. Advanced Technology Program’s Intramural Research Initiative, London: Routledge, 2005.

  

bWhile economic theory is formulated on what is called ‘revealed preferences,’—meaning individuals and firms reveal how they value scarce resources by how they allocate those resources within a market framework—quite often, expressed preferences are a better source of information, especially from an evaluation perspective. Strict adherence to a revealed preference paradigm could lead to misguided policy conclusions because the paradigm assumes that all policy choices are known and understood at the time that an individual or firm reveals its preferences and that all relevant markets for such preferences are operational. See Gregory G. Dess and Donald W. Beard, “Dimensions of Organizational Task Environments,” Administrative Science Quarterly 29 (1984):52–73, and Albert N. Link and John T. Scott, Public Accountability: Evaluating Technology-Based Institutions, Norwell, MA: Kluwer Academic Publishers, 1998.

  

cLink and Scott, Evaluating Public Research Institutions: The U.S. Advanced Technology Program’s Intramural Research Initiative op. cit.

  

dData from the National Institutes of Health indicates that a survey taken two years later would reveal very substantial increases in both the percentage of firms reaching the market and the amount of sales per project. See National Research Council, An Assessment of the Small Business Innovation Research Program at the National Institutes of Health, Charles W. Wessner, ed., Washington, DC: The National Academies Press, forthcoming.

Suggested Citation:"3 Scope and Methodology." National Research Council. 2008. An Assessment of the SBIR Program at the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/11929.
×

panies, and one firm started by a professor-husband and entrepreneur-wife team. The case studies include two woman-owned companies, one of which is actively operated by a woman who is also a member of a minority group.

The sample includes companies that were able to commercialize a product very quickly and companies whose technologies are taking considerable time to commercialize. Commercialization strategies of these firms include licensing agreements, contract research, sales of product produced in-house, commercialization partnerships, as well as the sale of technology or the company itself to a larger company.

The companies’ annual revenues range from $2 million to about $24 million. This includes companies whose share of annual revenue contributed by the SBIR program and other government grants comprise as little as 4 percent and as much as 70 percent.

The ten companies selected for the case studies do not represent a random sample. The companies were selected to provide companies of different ages and size, pursuing different technologies, located in different parts of the country, with differing forms of ownership, and with some, although varying degrees of, commercial success. Some of the companies are university spin-offs; some are company spin-offs; some are neither. Some received many SBIR grants; some relatively few. Some continue to obtain a high percentage of their funding from government sources; others have reduced the percentage to low numbers. To illustrate different types of successful outcomes, six of the cases selected were identified by NSF as “stars.” In addition, all of these cases selected are revenue-earning SBIR-funded companies. The process for selecting the case studies is explained in detail in Appendix D.

Suggested Citation:"3 Scope and Methodology." National Research Council. 2008. An Assessment of the SBIR Program at the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/11929.
×
Page 41
Suggested Citation:"3 Scope and Methodology." National Research Council. 2008. An Assessment of the SBIR Program at the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/11929.
×
Page 42
Suggested Citation:"3 Scope and Methodology." National Research Council. 2008. An Assessment of the SBIR Program at the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/11929.
×
Page 43
Suggested Citation:"3 Scope and Methodology." National Research Council. 2008. An Assessment of the SBIR Program at the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/11929.
×
Page 44
Suggested Citation:"3 Scope and Methodology." National Research Council. 2008. An Assessment of the SBIR Program at the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/11929.
×
Page 45
Suggested Citation:"3 Scope and Methodology." National Research Council. 2008. An Assessment of the SBIR Program at the National Science Foundation. Washington, DC: The National Academies Press. doi: 10.17226/11929.
×
Page 46
Next: 4 Overview of the NSF SBIR Program »
An Assessment of the SBIR Program at the National Science Foundation Get This Book
×
Buy Hardback | $99.00 Buy Ebook | $79.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The Small Business Innovation Research (SBIR) program is one of the largest examples of U.S. public-private partnerships. Founded in 1982, SBIR was designed to encourage small business to develop new processes and products and to provide quality research in support of the many missions of the U.S. government, including health, energy, the environment, and national defense. In response to a request from the U.S. Congress, the National Research Council assessed SBIR as administered by the five federal agencies that together make up 96 percent of program expenditures.

This book, one of six in the series, reports on the SBIR program at the National Science Foundation. The study finds that the SBIR program is sound in concept and effective in practice, but that it can also be improved. Currently, the program is delivering results that meet most of the congressional objectives, including stimulating technological innovation, increasing private-sector commercialization of innovations, using small businesses to meet federal research and development needs, and fostering participation by minority and disadvantaged persons. The book suggests ways in which the program can improve operations, continue to increase private-sector commercialization, and improve participation by women and minorities.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!