. "4. Developing Operational Definitions and Concepts." An Assessment of the Small Business Innovation Research Program: Project Methodology. Washington, DC: The National Academies Press, 2004.
The following HTML text is provided to enhance online
readability. Many aspects of typography translate only awkwardly to HTML.
Please use the page image
as the authoritative form to ensure accuracy.
An Assessement of the Small Business Innovation Research Program: Project Methodology
4.Developing Operational Definitions and Concepts
The study will identify core operational terms and concepts in advance of full development of the methodology. The following represents an initial identification of some terms and concepts:
The quality of SBIR research
Quality is a relative concept by definition, so an assessment of SBIR research quality must compare it to the quality of other research.26 Quality is also subjective, so the realization of value may depend on its perceived utility. The principal comparison here will be with other extra-mural research funded by the same agencies. The question of whether comparisons should focus only on R&D by other small businesses is yet to be addressed; this decision may be made on an agency-by-agency basis.27
SBIR’s value to agency missions
Given that agency missions and their associated sub-unit objectives differ substantially among, (and even within) agencies, the issue of SBIR’s value to agency missions will be addressed largely in the context of individual agency analysis. While a more generic set of answers would be helpful, it will be important to emphasize the challenges posed by multiple agencies with multiple missions, executed by multiple subunits. For example, some agencies, such as DoD and NASA, are “procurement agencies,” seeking tools for the nation’s military, while others, such as NSF and NIH, are not. These different goals may change the agency’s vision of SBIR’s role quite fundamentally. Generic mission elements include:
Technology needs (i.e., agency-identified technology gaps, such as a missing vaccine delivery system identified as a priority by NIH28);
Procurement needs (i.e., technologies that the agency needs for its own internal use, e.g., optical advances for smart weapons at DoD).
Expansion or commercialization of knowledge in agency’s field of stewardship (e.g., funding in relatively broad sub-fields of information technology at NSF);29
Technology transfer (i.e., promoting the adoption of agency-developed technology by others). Technology may become available to the sponsoring agency and others through a variety of paths. These include
Procurement from supply chain providers,
Purchase by the agency on the open market via successful commercialization by the SBIR firm (e.g., purchase of DoD-R&D-supported advanced sonar equipment),
Use by others of the technology whose development is sponsored by the agency and made available through such means as licensing, partnership arrangements, or by purchase on the open market. (e.g., a power plant may adopt technology available on the market and fostered by DoE’s SBIR, or a public health clinic may adopt a new vaccine delivery system available on the market and fostered by NIH’s SBIR).
To address agency-specific missions (e.g., national defense at DoD, health at NIH, energy at DoE), the Committee will closely consult with agency staff to develop operational definitions of success--in some cases at the level of sub-units (e.g., individual NIH institutes and centers.) Some overlay will likely occur (e.g., defense-health needs.)
Finally, agencies will undoubtedly have their own conceptions of how their SBIR program is judged in relation to their missions, and it is possible (perhaps likely) that some of these views will not fit well in the areas listed. The Committee is sensitive to these distinctions and differences, and will articulate these concepts at an early stage.
26
See K. Buchholz. "Criteria for the analysis of scientific quality," Scientometrics 32 (2), 1995:195-218.
27
See M. Brown, T.R. Curlee and S.R. Elliott, “Evaluating Technology Innovation Programs: The Use of Comparison Groups to Identify Impacts,” Research Policy, 24, 1995.
28
See for example, “Micromachined Ultrasound Ejector Arrays For Aerosol-Based Pulmonary Vaccine Delivery.” Response to SBIR Proposal PHS 2001 NIP Topic 009, Technologies to Overcome the Drawbacks of Needles and Syringes Contract No: 200-2001-00112
29
See, for example, Maryann Feldman and Maryellen Kelley, “Leveraging Research and Development: The Impact of the Advanced Technology Program,” in National Research Council, The Advanced Technology Program, C. Wessner, ed. Washington, D.C.: National Academy Press, 2001, for a potential model of how an agency can use SBIR to help foster specific areas of technical expertise. The paper, however, does not address the extent to which this is a conscious goal of DoD.
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole. Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 12
4. Developing Operational Definitions and Concepts
The study will identify core operational terms and concepts in advance of full development of the methodology. The
following represents an initial identification of some terms and concepts:
The quality of SBIR research
Quality is a relative concept by definition, so an assessment of SBIR research quality must compare it to the quality
of other research.26 Quality is also subjective, so the realization of value may depend on its perceived utility. The
principal comparison here will be with other extra-mural research funded by the same agencies. The question of
whether comparisons should focus only on R&D by other small businesses is yet to be addressed; this decision may
be made on an agency-by-agency basis.27
SBIR’s value to agency missions
Given that agency missions and their associated sub-unit objectives differ substantially among, (and even within)
agencies, the issue of SBIR’s value to agency missions will be addressed largely in the context of individual agency
analysis. While a more generic set of answers would be helpful, it will be important to emphasize the challenges
posed by multiple agencies with multiple missions, executed by multiple subunits. For example, some agencies, such
as DoD and NASA, are “procurement agencies,” seeking tools for the nation’s military, while others, such as NSF and
NIH, are not. These different goals may change the agency’s vision of SBIR’s role quite fundamentally.
Generic mission elements include:
• Technology needs (i.e., agency-identified technology gaps, such as a missing vaccine delivery system
identified as a priority by NIH28);
• Procurement needs (i.e., technologies that the agency needs for its own internal use, e.g., optical
advances for smart weapons at DoD).
• Expansion or commercialization of knowledge in agency’s field of stewardship (e.g., funding in
relatively broad sub-fields of information technology at NSF);29
• Technology transfer (i.e., promoting the adoption of agency-developed technology by others).
Technology may become available to the sponsoring agency and others through a variety of paths. These
include
Procurement from supply chain providers,
o
Purchase by the agency on the open market via successful commercialization by the SBIR firm
o
(e.g., purchase of DoD-R&D-supported advanced sonar equipment),
Use by others of the technology whose development is sponsored by the agency and made
o
available through such means as licensing, partnership arrangements, or by purchase on the open
market. (e.g., a power plant may adopt technology available on the market and fostered by DoE’s
SBIR, or a public health clinic may adopt a new vaccine delivery system available on the market
and fostered by NIH’s SBIR).
To address agency-specific missions (e.g., national defense at DoD, health at NIH, energy at DoE), the Committee
will closely consult with agency staff to develop operational definitions of success--in some cases at the level of sub-
units (e.g., individual NIH institutes and centers.) Some overlay will likely occur (e.g., defense-health needs.)
Finally, agencies will undoubtedly have their own conceptions of how their SBIR program is judged in relation to their
missions, and it is possible (perhaps likely) that some of these views will not fit well in the areas listed. The
Committee is sensitive to these distinctions and differences, and will articulate these concepts at an early stage.
26
See K. Buchholz. "Criteria for the analysis of scientific quality," Scientometrics 32 (2), 1995:195-218.
27
See M. Brown, T.R. Curlee and S.R. Elliott, “Evaluating Technology Innovation Programs: The Use of Comparison Groups to
Identify Impacts,” Research Policy, 24, 1995.
28
See for example, “Micromachined Ultrasound Ejector Arrays For Aerosol-Based Pulmonary Vaccine Delivery.” Response to SBIR
Proposal PHS 2001 NIP Topic 009, Technologies to Overcome the Drawbacks of Needles and Syringes Contract No: 200-2001-00112
29
See, for example, Maryann Feldman and Maryellen Kelley, “Leveraging Research and Development: The Impact of the Advanced
Technology Program,” in National Research Council, The Advanced Technology Program, C. Wessner, ed. Washington, D.C.:
National Academy Press, 2001, for a potential model of how an agency can use SBIR to help foster specific areas of technical
expertise. The paper, however, does not address the extent to which this is a conscious goal of DoD.
12
OCR for page 13
The extent of commercialization
SBIR is charged with supporting the commercialization of technologies developed with federal government support.
In many agencies, this requirement is articulated as a focus on the “commercialization” of SBIR supported research.30
At the simplest level, commercialization means, “reaching the market,” which some agency managers interpret as
“first sale”: the first sale of a product in the market place, whether to public or private sector clients.31 This
definition is certainly practical and defensible. However, it risks missing significant components of commercialization
that do not result in a discrete sale. At the same time, it also fails to provide any guidance on how to evaluate the
scale of commercialization, which is critical to assessing the degree to which SBIR programs successfully encourage
commercialization: the sale of a single widget is not the same as playing a critical role in the original development of
Qualcomm’s cell-phone technology.
Thus, the Committee’s assessment of commercialization will require working operational definitions for a number of
components. These include:
• Sales—what constitutes a sale?
• Application—how is the product used? For example, products like software are re-used repeatedly.
• Measuring scale—over what interval is the impact to be measured. (e.g., Qualcomm’s SBIR grant was
by all accounts very important for the company. The question arises as to how long the dollar value of
Qualcomm’s wireless related sales, stemming from its original SBIR grant, should be counted.)32
• Licensing—how should commercial sales generated by third party licensees of the original technology
be counted. Is the licensing revenue from the licensee to be counted, or the sales of that technology
by the licensee— (or both)?
• Complex sales—technologies are often sold as bundles with other technologies (auto engines with
mufflers for example). Given this, how is the share of the total sales value attributable to the
technology that received SBIR funding to be defined?
• Lags—some technologies reach market rapidly, but others can take 10 years or more. What is an
appropriate discount rate and timeframe to measure award impact?
Metrics for assessing commercialization can be elusive. Notably, one cannot easily calculate the full value of
developed “enabling technology” that can be used across industries. Also elusive is the value of material that
enables a commercial service. In such cases, a qualitative approach to “commercialization” will need to be
employed.
While the theoretical concept of additionality will be of some relevance to these questions, practicalities must govern,
and the availability of data will substantially shape the Committee’s approach in this area. This is particularly the
case where useful data must be gathered from thousands of companies, often at very considerable expense in
dollars and time. 33
The NRC study will resolve these very practical questions by the early stages of the study’s second phase. The
Committee plans to adapt, where appropriate, definitions and approaches used in the Fast Track study for the
current study.34
Broad economic effects
SBIR programs may generate a wide range of economic effects. While some of these may be best considered in a
national context, others fall more directly on participating firms and on the agencies themselves. The Committee will
consider these possible benefits and costs in terms of the level of incidence.
30
A key objective of the 1982 Small Business Innovation Development Act is to increase private sector commercialization derived
from federal research and development. The role of SBIR in stimulating commercialization was cited as a justification in the
reauthorization of the Act in 1992: that SBIR “has effectively stimulated the commercialization of technology development through
federal research and development, benefiting both the public and private sectors of the Nation.”
31
For analysis of observed variations in timelines for commercialization, see NISTIR 6917
”Different Timelines for Different Technologies: Evidence from the Advanced Technology Program” at
http://www.atp.nist.gov/eao/ir-6917/chapt5.htm
32
For a profile of Qualcomm, see http://www.inknowvation.com/cgi-bin/db4/Qualcomm_Profile.html
33
Buisseret, T.J., Cameron, H., and Georghiou, L. (1995) “What difference does it make? Additionality in the public support of R&D
in large firms”, International Journal of Technology Management, Vol.10, Nos. 4/5/6 pp. 587-600. See also, Luke Georghiou,
“Impact and Additionality of Innovation Policy,” Paper presented at the Six Countries Programme on Innovation, Spring 2002,
Brussels.
34
See National Research Council, SBIR: An Assessment of the Department of Defense Fast Track Initiative, 2000, op. cit.
13
OCR for page 14
Participating firms
Economic effects on firms include some or all of the following elements:
Revenue from sale or adoption of SBIR-developed products, services, or processes (this tracks quite
•
closely but not 100 percent with commercialization)
• Changes in the firm’s access to capital, including ways in which SBIR awards have helped (or
hindered) recipient companies access capital markets
• Change in firm viability and sustainability, including how the SBIR program helped bridge the
gaps between these interrelated stages of the innovation process;35
Conception
o
Innovation
o
Product development
o
Entry into market
o
• Changes in the propensity to partner and the nature of the partnerships
The impact of SBIR on the frequency with which companies develop partnerships
o
The nature of the partnerships—are they public or private partners?
o
• Enhanced firm growth, productivity, profitability
Change in employment and capitalization36
o
Change in firm productivity
o
Change in profits
o
• Intellectual property developed by the firm
Work in this section will follow closely on the Fast Track study model, seeking to identify ways
o
in which the recipient firms were affected in the areas listed.37
The agencies
Effects on the agencies include the following:
• Effects on mission support
• Effects on agency research efficiency, including whether --
SBIR has helped to generate technologies that agencies might not otherwise have developed
o
in the same timeframe without the program
SBIR is an effective way for agencies to fund competitive research, presumably compared to
o
non-SBIR research funding for small scale requirements
There are significant benefits to agency missions from the specific effort of SBIR to capture
o
research by small firms. Benchmark numbers for small business contributions to agency
research programs before SBIR (pre-1983), and outside SBIR (other programs?) may be
needed.
Research efficiency implies a review of the returns to the agency from SBIR investment vs. other
research investment. It is important to acknowledge, however, that this analysis will likely not be
based on hard rate of return analysis, because the data necessary for such analysis is unlikely to be
available at the agency level.
• Effects on agency procurement efficiency
35
Though commonly conceived as a linear process, innovation is characterized by significant complexity. For a discussion of this
complexity, see, National Research Council, The Small Business Innovation Research Program: Program Diversity and Assessment
Challenges, C. Wessner, ed. Washington, D.C.: National Academies Press, 2004.
36
The employment effects of research are most often indirect—through the application or commercialization of the research.
37
Ibid.
14
OCR for page 15
Effects on society
• Social returns refer to the returns to society at large, including private returns and spillover effects.
Using the Link/Scott approach from Fast Track as a model is expected to help us conceptualize our
approach to this broad effect.38 Other approaches may also be useful.
• Small business support refers to the positive social externalities associated with a vibrant small
business sector, including community cohesiveness and improvements to life made possible new
products. Measuring the impact of program support for small business is a major objective,39 given
that support encourages the commercialization of public investment in R&D, the achievement of
national missions, and the encouragement of small firm growth.
• Training in both business development and in technology and innovation The Audtresch/ J.
Weigand/ C. Weigand Fast Track paper provides a good methodological basis for addressing
“training.”40
Non-economic benefits
While it is possible to view almost all non-economic effects through the lens of economic analysis, pushing all effects
to economic measurement is usually not feasible and may not be appropriate. Certain effects have been specifically
defined as positive outcomes by Congress, regardless of whether they have any measurable impact on economic
well-being. This section addresses non-economic benefits, which will in turn have non-economic metrics attached to
them, discussed later in this paper.
Knowledge benefits
The missions of several agencies explicitly state the requirement of advancing knowledge in the relevant field. For
the SBIR program, this requirement can be viewed from two distinct but complementary perspectives:
Intellectual property,41 which is governed by a set of legal definitions, and is susceptible to close
•
measurement via analysis of patent filings and other largely quantitative assessment strategies.42
Intellectual property rights are generally used to convert knowledge to property for commercial benefit of
the owner. At the same time, mechanisms of intellectual property can help to disseminate knowledge to
others. A patent, for example, gives the holder exclusive rights, but provides information to others.
Intellectual property also includes “trade secrets.” In many cases, the “know how” that firms keep
proprietary may be the most important intellectual property produced by the research. These may be less
susceptible to measurement.
• Non-property knowledge is much less well defined but nonetheless of great importance. Non-property
knowledge ranges from formal activities (e.g., papers published in refereed journals, and seminars) to very
informal activities (e.g., discussions among researchers and worker mobility). Many relevant concepts are
discussed in the literature on human capital. Non-property knowledge is related to education and training
and encompasses network capital and tacit expertise that an engineer or scientist may possess.
38
See Link, A. N. and Scott, J. T. Public Accountability: Evaluating Technology-Based Institutions, Boston: Kluwer Academic
Publishers, 1998. Link and Scott use published agency data and interviews to determine key indicators, including private hurdle
rates, additional anticipated development time after Phase II, additional cost, life of the commercialized technology, and proportion
of value appropriated by firm. These allow estimates of social and private returns. Link and Scott indicate that the rates of return
on the SBIR Phase II investment for Fast Track were 84 percent for society and 25 percent for private investors.
39
See also section 4.5 below.
40
See David Audretsch, Jeurgen Weigand and Claudia Weigand, “The Impact of the SBIR on creating entrepreneurial behavior,”
Economic Development Quarterly Vol. 16, No. 1, February 2002, pp. 32-38. Audretsch/Weigand/Weigand identify interesting
spillover effects of SBIR grants on non-recipient scientists and engineers, in terms of career paths, entrepreneurial activities and
their timing, etc.
41
Intellectual property is divided into two categories: Industrial property, which includes inventions (patents), trademarks, industrial
designs, and geographic indications of source; and Copyright, which includes literary and artistic works such as novels, poems and
plays, films, musical works, artistic works such as drawings, paintings, photographs and sculptures, and architectural designs.
Rights related to copyright include those of performing artists in their performances, producers of phonograms in their recordings,
and those of broadcasters in their radio and television programs. See http://www.wipo.org/about-ip/en/
42
For a summary of such measures, see Table 1 in European Commission, Directorate General Enterprises “Enterprises and SME”
Programme, “European Trend Chart on Innovation,” June 2000. Accessed at
http://trendchart.cordis.lu/Reports/Documents/Innovation_and_IPR_June2000.pdf
15
OCR for page 16
Other potential non-economic benefits
• Environmental impacts
• Safety
• Quality of life
Trends in agency funding for small business
For this study, “small business funding” will be defined as synonymous with SBIR. A definition of “small” is needed.
The SBIR definition (fewer than 500 employees) is quite broad.43 Dividing firm participants into size subcategories
may be advantageous. We plan a breakdown of small firms by size, taking into account existing SBA classifications
and based on natural divisions as emerge from the data.
The agencies have considerable discretion in defining which agency expenditures and disbursements they consider to
be R&D, and thus subject to the percentage requirements of the SBIR set aside. Small firms also receive R&D
funding directly from the agencies outside the SBIR program, and receive subcontracts for R&D from primes or other
subcontractors, whose original funding source was federal R&D. Since in some cases the prime or intermediate
contractor may also be a small business, there is an opportunity for double counting as well as for undercounting. It
is important to keep in mind that the congressional intent was to increase the amount of federal R&D funding
ultimately reaching small businesses.
Small businesses, in many cases, cannot take on more R&D funding, as they do not have the expert staff, or the
culture to do R&D. Thus, there might conceivably be a sort of saturation effect. The issue of absorptive capacity
also occurs in the case of fast moving high tech firms, which may not willing to risk the overhead and delay involved
in seeking federal funds at all. In the present study, saturation effects can be examined in part by investigating the
relationship between the growth of grant-program funding and the growth of grant-program applications. Insights
into the impacts of expansions in grant funding on small-business response capacity and on research quality may be
gained by analyzing ATP’s experience between 1993 and 1994, based on changes in reviewer technical scores and
small business application rates as the program was expanded dramatically between 1993 and 1994.
Best practices and procedures in operating SBIR programs
Issues related to administrative process, both within agencies and across agencies, will be defined over the course of
the first phase of the NRC study. Areas to be addressed may include:
• Outreach
• Topic development
• Application procedures and timelines
• Project monitoring
• Agency management funding
• Project funding limitations
• Bridge funding
• Post SBIR Phase II support
43
According to the Small Business Administration, a small business is a concern that is organized for profit, with a place of business
in the United States, and which operates primarily within the United States or makes a significant contribution to the U.S. economy
through payment of taxes or use of American products, materials, or labor. Further, the concern cannot be dominant in its field, on
a national basis. Finally, the concern must meet the numerical small business size standard for its industry. SBA has established a
size standard for most industries in the U.S. economy. The most common size standards are 500 employees for most
manufacturing and mining industries, 100 employees for all wholesale trade industries, $6 million for most retail and service
industries, $28.5 million for most general & heavy construction industries, $12 million for all special trade contractors, and
$0.75 million for most agricultural industries.
16
