National Academies Press: OpenBook
« Previous: Front Matter
Suggested Citation:"EXECUTIVE SUMMARY." National Research Council. 2004. An Assessment of the Small Business Innovation Research Program: Project Methodology. Washington, DC: The National Academies Press. doi: 10.17226/11097.
Page 1
Suggested Citation:"EXECUTIVE SUMMARY." National Research Council. 2004. An Assessment of the Small Business Innovation Research Program: Project Methodology. Washington, DC: The National Academies Press. doi: 10.17226/11097.
Page 2
Suggested Citation:"EXECUTIVE SUMMARY." National Research Council. 2004. An Assessment of the Small Business Innovation Research Program: Project Methodology. Washington, DC: The National Academies Press. doi: 10.17226/11097.
Page 3

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Executive Summary The Small Business Reauthorization Act of 2000, H.R. 5667, Section 108, enacted in Public Law 106-554, requests that the National Research Council undertake a review of the of the Small Business Innovation Research program (SBIR) at the five federal agencies with SBIR programs with budgets in excess of $50 million. These five agencies, in order of program size, are the Department of Defense, the National Institutes of Health, the National Aeronautics and Space Administration, the Department of Energy, and the National Science Foundation. The Study Charge This study will review the SBIR program at the five agencies with regard to parameters such as the quality of the research projects being conducted under the SBIR program, the commercialization of the research, and the program's contribution to accomplishing agency missions. To the extent possible, the evaluation will include estimates of the benefits (both economic and non-economic) achieved by the SBIR program. The study will also examine broader policy issues associated with public-private collaborations for technology development and government support for high technology innovation. The project will encourage cross-fertilization among program managers, agency officials, and participants by convening national experts from industry, academia, and the public sector to review and discuss research findings. Where appropriate, operational improvements to the program will be considered. The Objectives of the Study The objectives of the study are to: Satisfy the Congressional mandate for an objective, external assessment of the program; Provide an empirical analysis of the operations of the SBIR program, in particular rates and sources of commercialization, for agency officials and program managers; Address research questions relevant to the program's operation and evaluation derived from the legislation and that emerge in the course of the study; Develop a rigorous assessment of the program and contribute to Congressional understanding of its multiple objectives, measurement issues, operational challenges, and contributions as described in the legislation. Focus of the Evaluation Following the passage of HR 5667 in December 2000, extensive discussions were held between the NRC and the five leading agencies regarding the scope and nature required to fulfill the Congressional mandate. Agreement on the terms of the study was reached in December 2001, and the requisite funding for the Academies to begin the study was received in September 2002. The study was officially launched on 1 October 2002. The Memorandum of Understanding between the NRC and the agencies reflects the Congressional mandate by specifying a particular focus for the evaluation on four aspects of the SBIR Program: 1. Commercialization. Congress established the SBIR program partly to support commercialization of Federal research. The agencies have in general interpreted this to mean support for research activities, which could result in successful commercialization, measured in different ways, while also meeting other objectives. 2. Mission support. Congress has also mandated that SBIR programs should support the mission of the funding agency. Of course, each agency has a different mission, which means that, at least in part, different indicators or metrics will be needed. Indeed, initial research indicates that there are very significant differences in this area among agencies that fund high tech research in order to eventually purchase the outputs from goods and services that may emerge from it (DoD, NASA, parts of DoE) and those that do not (NSF, NIH, parts of DoE). This basic difference suggests the need for quite different research strategies, but, in both cases, the study seeks to establish the extent to which SBIR programs meet this component of the Congressional mandate. 3. Knowledge base. All federal research includes the objective of expanding the nation’s knowledge base. SBIR programs are also charged with this objective, which appears to be doubly important for the non-procuring agencies. For these non-procuring agencies, a substantial part of the agency mission could also be described as to the expansion of the knowledge base, through intermediate and final products. 1

4. Program management. The charge to the Committee includes the provision of recommendations for improving the SBIR program (although not for assessing its continued existence). That charge requires review and assessment of how each agency SBIR program operates, and—where possible—the identification of best practices and possible improvements. Limits of the Committee Charge The objective of the study is not to consider if SBIR should exist or not—Congress has already decided affirmatively on this question. Rather, the NRC Committee conducting this study is charged with providing assessment-based findings of the benefits and costs of SBIR (described in the Objectives section above) to improve public understanding of the program, as well as recommendations to improve the program’s effectiveness. It is also important to note that, in accordance with the Memorandum of Understanding and the Congressional mandate, the study will not seek to compare the value of one area with other areas; this task is the prerogative of the Congress and the Administration acting through the agencies. Instead, the study is concerned with the effective review of each area. A Two-Phase Study Structure The project is divided into two phases. Phase I has focused on data collection and the development of the methodology. Per the agreement with the agencies as outlined in the Memorandum of Understanding,3 this Methodology was submitted to an intensive Academy review process, involving 12 reviewers with recognized expertise in economics, statistics, program evaluation, survey methodology, innovation policy, federal R&D programs, and both large and small high-tech firms. Extensive revisions and elaborations were required as a result of this review and they are now reflected in this document. The second phase of the study will now implement the research methodology developed in Phase I. Phase I included an initial symposium for the program as a whole, followed by a number of committee meetings and a series of workshops to address the specific features of each agency's program. This phase has focused on the development of survey instruments, case study templates, and related research to the extent possible. Additional details regarding the study methodologies to be used have been deferred in some cases because they cannot be defined precisely until some initial Phase II work has been completed. At the conclusion of Phase I, the overall methodology and evaluation tools will be submitted for review. In Phase II, research papers on general topics will be commissioned, and preliminary results of field research will be assessed and cross-checked. A symposium will be convened to discuss publicly the results of the research, and final reports will be prepared for each agency and for the program as a whole. Summary of Methods to be Used The purpose of this document is to describe the methodological approaches developed under Phase I of the study. They build from the precedents established in several key studies already undertaken to evaluate various aspects of the SBIR. These studies have been successful because they identified the need for utilizing not just a single methodological approach, but rather a broad spectrum of approaches, in order to evaluate the SBIR from a number of different perspectives and criteria. This diversity and flexibility in methodological approach are particularly appropriate given the heterogeneity of goals and procedures across the five agencies involved in the evaluation. Consequently, this document suggests a broad framework for methodological approaches that can serve to guide the research team when evaluating each particular agency in terms of the four criteria stated above. Table 1 illustrates some key assessment parameters and related measures to be considered in this study.4 3 See Annex C in this volume. 4 See also Annex F in this volume. 2

TABLE 1: OVERVIEW OF APPROACH TO SBIR PROGRAM ASSESSMENT SBIR Assessment Quality of Commercialization Small Business Use of Small Parameters Research of SBIR Funded Innovation/ Businesses to Research/ Growth Advance Agency Economic and non- Missions Economic benefits Questions How does the quality What is the overall How to broaden How to increase of SBIR funded economic impact of participation and agency uptake while research compare SBIR funded replenish contractors? continuing to support with that of other research? What is the link high risk research government funded What fraction of that between SBIR and R&D? impact is attributable state/regional to SBIR funding? programs? Measures Peer review scores Sales; follow up Patent counts and Agency procurement Publication counts funding; progress; other IP / of products resulting Citation analysis IPO employment growth, from SBIR work number of new technology firms Tools° Case Studies, Agency Phase II surveys, Phase I and Phase II Program Manager Program Studies, Program Manager surveys, Case Surveys, Case Study of Repeat Surveys, Case Studies, Study of Studies, Agency Winners, Bibliometric Studies, Study of Repeat Winners, Program Studies, Analysis Repeat Winners Bibiometric Analysis Study of Repeat Winners Key Research Difficulty of Skew of returns; Measures of actual Major interagency Challenges measuring quality Significant success and failure at differences in use of and of identifying interagency and the project and firm SBIR to meet agency proper reference inter-industry level; Relationship of missions group differences federal and state programs in this context ° Supplementary tools may be developed and used as needed. Multiple Methodologies Over the iterative development of the study’s methodology, it became clear that no single research methodology would suffice to assess a program as differentiated as SBIR—one with multiple objectives, distinctive agency missions, and varied participants (ranging from small start-ups to relatively large, well-established companies, with product cycles ranging from months to decades). Instead, a complement of methodological tools has been crafted to address different facets of the program’s operation. These tools are firmly grounded in economics and, as noted, draw from the experience of successful approaches pioneered by previous NRC studies of SBIR. They will necessarily have to be implemented in a flexible manner, with additional approaches to be drafted as new research challenges emerge. This document is, in this sense, a working draft, reflecting the current state of the Research Team and Committee discussions. It represents the Committee’s considered understanding of the tasks at hand, and methodological tools that can be applied to address these tasks. The elaborated methodologies are, thus, not exclusive, precluding the adoption of other tools and approaches; nor are they definitive, representing a fixed and final statement. Instead, the document provides a summary of current thinking on the project, as it has evolved from the discussions of the Research Team and the Steering Committee, as well as other interested parties. Despite these necessary limitations, this document constitutes a clear statement of the research goals and the tools the Committee plans to use to address them. 3

Next: 1. Introduction »
An Assessment of the Small Business Innovation Research Program: Project Methodology Get This Book
Buy Paperback | $43.00 Buy Ebook | $34.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

In response to a Congressional mandate, the National Research Council conducted a review of the SBIR program at the five federal agencies with SBIR programs with budgets in excess of $100 million (DOD, NIH, NASA, DOE, and NSF). The project was designed to answer questions of program operation and effectiveness, including the quality of the research projects being conducted under the SBIR program, the commercialization of the research, and the program's contribution to accomplishing agency missions. This report describes the proposed methodology for the project, identifying how the following tasks will be carried out: 1) collecting and analyzing agency databases and studies; 2) surveying firms and agencies; 3) conducting case studies organized around a common template; and 4) reviewing and analyzing survey and case study results and program accomplishments. Given the heterogeneity of goals and procedures across the five agencies involved, a broad spectrum of evaluative approaches is recommended.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook,'s online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!