. "EXECUTIVE SUMMARY." An Assessment of the Small Business Innovation Research Program: Project Methodology. Washington, DC: The National Academies Press, 2004.
The following HTML text is provided to enhance online
readability. Many aspects of typography translate only awkwardly to HTML.
Please use the page image
as the authoritative form to ensure accuracy.
An Assessement of the Small Business Innovation Research Program: Project Methodology
Executive Summary
The Small Business Reauthorization Act of 2000, H.R. 5667, Section 108, enacted in Public Law 106-554, requests that the National Research Council undertake a review of the of the Small Business Innovation Research program (SBIR) at the five federal agencies with SBIR programs with budgets in excess of $50 million. These five agencies, in order of program size, are the Department of Defense, the National Institutes of Health, the National Aeronautics and Space Administration, the Department of Energy, and the National Science Foundation.
The Study Charge
This study will review the SBIR program at the five agencies with regard to parameters such as the quality of the research projects being conducted under the SBIR program, the commercialization of the research, and the program's contribution to accomplishing agency missions. To the extent possible, the evaluation will include estimates of the benefits (both economic and non-economic) achieved by the SBIR program. The study will also examine broader policy issues associated with public-private collaborations for technology development and government support for high technology innovation. The project will encourage cross-fertilization among program managers, agency officials, and participants by convening national experts from industry, academia, and the public sector to review and discuss research findings. Where appropriate, operational improvements to the program will be considered.
The Objectives of the Study
The objectives of the study are to:
Satisfy the Congressional mandate for an objective, external assessment of the program;
Provide an empirical analysis of the operations of the SBIR program, in particular rates and sources of commercialization, for agency officials and program managers;
Address research questions relevant to the program's operation and evaluation derived from the legislation and that emerge in the course of the study;
Develop a rigorous assessment of the program and contribute to Congressional understanding of its multiple objectives, measurement issues, operational challenges, and contributions as described in the legislation.
Focus of the Evaluation
Following the passage of HR 5667 in December 2000, extensive discussions were held between the NRC and the five leading agencies regarding the scope and nature required to fulfill the Congressional mandate. Agreement on the terms of the study was reached in December 2001, and the requisite funding for the Academies to begin the study was received in September 2002. The study was officially launched on 1 October 2002. The Memorandum of Understanding between the NRC and the agencies reflects the Congressional mandate by specifying a particular focus for the evaluation on four aspects of the SBIR Program:
Commercialization. Congress established the SBIR program partly to support commercialization of Federal research. The agencies have in general interpreted this to mean support for research activities, which could result in successful commercialization, measured in different ways, while also meeting other objectives.
Mission support. Congress has also mandated that SBIR programs should support the mission of the funding agency. Of course, each agency has a different mission, which means that, at least in part, different indicators or metrics will be needed. Indeed, initial research indicates that there are very significant differences in this area among agencies that fund high tech research in order to eventually purchase the outputs from goods and services that may emerge from it (DoD, NASA, parts of DoE) and those that do not (NSF, NIH, parts of DoE). This basic difference suggests the need for quite different research strategies, but, in both cases, the study seeks to establish the extent to which SBIR programs meet this component of the Congressional mandate.
Knowledge base. All federal research includes the objective of expanding the nation’s knowledge base. SBIR programs are also charged with this objective, which appears to be doubly important for the non-procuring agencies. For these non-procuring agencies, a substantial part of the agency mission could also be described as to the expansion of the knowledge base, through intermediate and final products.
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole. Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 1
Executive Summary
The Small Business Reauthorization Act of 2000, H.R. 5667, Section 108, enacted in Public Law 106-554, requests
that the National Research Council undertake a review of the of the Small Business Innovation Research program
(SBIR) at the five federal agencies with SBIR programs with budgets in excess of $50 million. These five agencies, in
order of program size, are the Department of Defense, the National Institutes of Health, the National Aeronautics and
Space Administration, the Department of Energy, and the National Science Foundation.
The Study Charge
This study will review the SBIR program at the five agencies with regard to parameters such as the quality of the
research projects being conducted under the SBIR program, the commercialization of the research, and the
program's contribution to accomplishing agency missions. To the extent possible, the evaluation will include
estimates of the benefits (both economic and non-economic) achieved by the SBIR program. The study will also
examine broader policy issues associated with public-private collaborations for technology development and
government support for high technology innovation. The project will encourage cross-fertilization among program
managers, agency officials, and participants by convening national experts from industry, academia, and the public
sector to review and discuss research findings. Where appropriate, operational improvements to the program will be
considered.
The Objectives of the Study
The objectives of the study are to:
Satisfy the Congressional mandate for an objective, external assessment of the program;
Provide an empirical analysis of the operations of the SBIR program, in particular rates and sources of
commercialization, for agency officials and program managers;
Address research questions relevant to the program's operation and evaluation derived from the legislation and
that emerge in the course of the study;
Develop a rigorous assessment of the program and contribute to Congressional understanding of its multiple
objectives, measurement issues, operational challenges, and contributions as described in the legislation.
Focus of the Evaluation
Following the passage of HR 5667 in December 2000, extensive discussions were held between the NRC and the five
leading agencies regarding the scope and nature required to fulfill the Congressional mandate. Agreement on the
terms of the study was reached in December 2001, and the requisite funding for the Academies to begin the study
was received in September 2002. The study was officially launched on 1 October 2002. The Memorandum of
Understanding between the NRC and the agencies reflects the Congressional mandate by specifying a particular
focus for the evaluation on four aspects of the SBIR Program:
1. Commercialization. Congress established the SBIR program partly to support commercialization of Federal
research. The agencies have in general interpreted this to mean support for research activities, which could
result in successful commercialization, measured in different ways, while also meeting other objectives.
2. Mission support. Congress has also mandated that SBIR programs should support the mission of the funding
agency. Of course, each agency has a different mission, which means that, at least in part, different indicators
or metrics will be needed. Indeed, initial research indicates that there are very significant differences in this
area among agencies that fund high tech research in order to eventually purchase the outputs from goods and
services that may emerge from it (DoD, NASA, parts of DoE) and those that do not (NSF, NIH, parts of DoE).
This basic difference suggests the need for quite different research strategies, but, in both cases, the study
seeks to establish the extent to which SBIR programs meet this component of the Congressional mandate.
3. Knowledge base. All federal research includes the objective of expanding the nation’s knowledge base. SBIR
programs are also charged with this objective, which appears to be doubly important for the non-procuring
agencies. For these non-procuring agencies, a substantial part of the agency mission could also be described as
to the expansion of the knowledge base, through intermediate and final products.
1
OCR for page 2
4. Program management. The charge to the Committee includes the provision of recommendations for
improving the SBIR program (although not for assessing its continued existence). That charge requires review
and assessment of how each agency SBIR program operates, and—where possible—the identification of best
practices and possible improvements.
Limits of the Committee Charge
The objective of the study is not to consider if SBIR should exist or not—Congress has already decided affirmatively
on this question. Rather, the NRC Committee conducting this study is charged with providing assessment-based
findings of the benefits and costs of SBIR (described in the Objectives section above) to improve public
understanding of the program, as well as recommendations to improve the program’s effectiveness. It is also
important to note that, in accordance with the Memorandum of Understanding and the Congressional mandate, the
study will not seek to compare the value of one area with other areas; this task is the prerogative of the Congress
and the Administration acting through the agencies. Instead, the study is concerned with the effective review of
each area.
A Two-Phase Study Structure
The project is divided into two phases. Phase I has focused on data collection and the development of the
methodology. Per the agreement with the agencies as outlined in the Memorandum of Understanding,3 this
Methodology was submitted to an intensive Academy review process, involving 12 reviewers with recognized
expertise in economics, statistics, program evaluation, survey methodology, innovation policy, federal R&D programs,
and both large and small high-tech firms.
Extensive revisions and elaborations were required as a result of this review and they are now reflected in this
document. The second phase of the study will now implement the research methodology developed in Phase I.
Phase I included an initial symposium for the program as a whole, followed by a number of committee meetings and
a series of workshops to address the specific features of each agency's program. This phase has focused on the
development of survey instruments, case study templates, and related research to the extent possible. Additional
details regarding the study methodologies to be used have been deferred in some cases because they cannot be
defined precisely until some initial Phase II work has been completed. At the conclusion of Phase I, the overall
methodology and evaluation tools will be submitted for review. In Phase II, research papers on general topics will be
commissioned, and preliminary results of field research will be assessed and cross-checked. A symposium will be
convened to discuss publicly the results of the research, and final reports will be prepared for each agency and for
the program as a whole.
Summary of Methods to be Used
The purpose of this document is to describe the methodological approaches developed under Phase I of the study.
They build from the precedents established in several key studies already undertaken to evaluate various aspects of
the SBIR. These studies have been successful because they identified the need for utilizing not just a single
methodological approach, but rather a broad spectrum of approaches, in order to evaluate the SBIR from a number
of different perspectives and criteria.
This diversity and flexibility in methodological approach are particularly appropriate given the heterogeneity of goals
and procedures across the five agencies involved in the evaluation. Consequently, this document suggests a broad
framework for methodological approaches that can serve to guide the research team when evaluating each particular
agency in terms of the four criteria stated above. Table 1 illustrates some key assessment parameters and related
measures to be considered in this study.4
3 See Annex C in this volume.
4
See also Annex F in this volume.
2
OCR for page 3
TABLE 1: OVERVIEW OF APPROACH TO SBIR PROGRAM ASSESSMENT
SBIR Assessment Quality of Commercialization Small Business Use of Small
Parameters Research of SBIR Funded Innovation/ Businesses to
Research/ Growth Advance Agency
Economic and non- Missions
Economic benefits
Questions How does the quality What is the overall How to broaden How to increase
of SBIR funded economic impact of participation and agency uptake while
research compare SBIR funded replenish contractors? continuing to support
with that of other research? What is the link high risk research
government funded What fraction of that between SBIR and
R&D? impact is attributable state/regional
to SBIR funding? programs?
Measures Peer review scores Sales; follow up Patent counts and Agency procurement
Publication counts funding; progress; other IP / of products resulting
Citation analysis IPO employment growth, from SBIR work
number of new
technology firms
Tools° Case Studies, Agency Phase II surveys, Phase I and Phase II Program Manager
Program Studies, Program Manager surveys, Case Surveys, Case
Study of Repeat Surveys, Case Studies, Study of Studies, Agency
Winners, Bibliometric Studies, Study of Repeat Winners, Program Studies,
Analysis Repeat Winners Bibiometric Analysis Study of Repeat
Winners
Key Research Difficulty of Skew of returns; Measures of actual Major interagency
Challenges measuring quality Significant success and failure at differences in use of
and of identifying interagency and the project and firm SBIR to meet agency
proper reference inter-industry level; Relationship of missions
group differences federal and state
programs in this
context
°
Supplementary tools may be developed and used as needed.
Multiple Methodologies
Over the iterative development of the study’s methodology, it became clear that no single research methodology
would suffice to assess a program as differentiated as SBIR—one with multiple objectives, distinctive agency
missions, and varied participants (ranging from small start-ups to relatively large, well-established companies, with
product cycles ranging from months to decades). Instead, a complement of methodological tools has been crafted to
address different facets of the program’s operation.
These tools are firmly grounded in economics and, as noted, draw from the experience of successful approaches
pioneered by previous NRC studies of SBIR. They will necessarily have to be implemented in a flexible manner, with
additional approaches to be drafted as new research challenges emerge. This document is, in this sense, a working
draft, reflecting the current state of the Research Team and Committee discussions. It represents the Committee’s
considered understanding of the tasks at hand, and methodological tools that can be applied to address these tasks.
The elaborated methodologies are, thus, not exclusive, precluding the adoption of other tools and approaches; nor
are they definitive, representing a fixed and final statement. Instead, the document provides a summary of current
thinking on the project, as it has evolved from the discussions of the Research Team and the Steering Committee, as
well as other interested parties. Despite these necessary limitations, this document constitutes a clear statement of
the research goals and the tools the Committee plans to use to address them.
3
