Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 93
Pane! V
Improving Assessment and Selection
INTRODUCTION
Dan Roos
Massachusetts Institute of Technology
Dr. Roos opened the session with the announcement that Herbert Kreitman
would substitute for Wendy Baldwin as the representative of the National Insti-
tutes of Health (NIH) on the panel. He then introduced Maryellen Kelley, serving
as a senior economist for the Advanced Technology Program (ATP) at the
National Institute of Standards and Technology while on leave from her post as
associate professor at the Heinz School of Public Policy and Management at
Carnegie Mellon University. Dr. Kelley discussed the lessons emerging from
ATP from two perspectives: (1) to what extent small businesses are involved in
the program, and (2) how ATP compares with the SBIR program from the view-
point of assessment, selection, and objectives.
LESSONS FROM THE ADVANCED TECHNOLOGY PROGRAM
Maryellen R. Kelley
NIST and Carnegie Mellon University
Differences between ATP and SBIR
Beginning with a description of ATP's operation, Dr. Kelley outlined fea-
tures of the program that make it unique in the federal government's technology
93
OCR for page 94
94
THE SMALL BUSINESS INNOVATION AND RESEARCH PROGRAM
policy portfolio. ATP is aimed at directly supporting R&D activities that have a
commercial orientation from their inception, unlike SBIR which pursues the goal
of commercialization through R&D that supports the mission of the SBIR-
member agency making the award. Like SBIR, as well as most government
agencies that are engaged in R&D, ATP funds activities that are undertaken by
private companies; in ATP's case, a cost-sharing requirement is designed to
leverage private R&D investment with the goal of achieving public benefits.
ATP is intended to support "enabling" technologies, which are characterized
by both high technical risk and the potential for substantial, widespread commer-
cial application. Technologies considered enabling can be:
.
· path-breaking those that promise to induce radical innovations;
· infrastructural those that support R&D, production, or businesses in
entire industries; or
multi-use those that either become platforms for multiple paths to
further technology development or are "generic" in that they have many
applications.
Rather than advancing a specific agency mission, the program's endeavors are
"industry driven." An annual general competition is open to proposals from all
fields of technology; in addition, there are numerous competitions each year
devoted to preselected areas of technology focus that have been identified by
ATP program managers in cooperation with industry.
Evaluating ATP
Study of the impact of the program, which was inaugurated in 1990, has been
limited to date because of the long-term nature of the projects it funds. In the
early stages, benefits that become apparent are largely in the generation of knowl-
edge knowledge that may, to some extent, also be of use to non-ATP firms and
for technologies other than those directly targeted by an award. In the long term,
knowledge spillovers are expected, as are other types of benefits that do not accrue
exclusively to those involved directly in the innovation process. In the case of
infrastructure-enabling technologies, other firms may learn about the technolo-
gies developed within ATP and build upon them. Also envisioned are market
spillovers: Knowledge that reaches beyond the ATP-awardee firms may allow
other firms to build cheaper and better products, eventually benefiting the
consumer.
"These are the ways in which the larger benefits of the program are realized
in the economy," explained Dr. Kelley. "That, in some sense, is the major justi-
fication for this type of effort: that it will have benefits beyond simply the private
returns to the individual firms."
OCR for page 95
PANEL V
.
.
95
Implications of ATP Evaluation for SBIR
ATP sees monitoring and evaluation as integral to an ongoing process of
refining the program so that spillover benefits can be captured more effectively.
Tracking projects as they progress, ATP' s management is able to consider impli-
cations for the program's strategy while at the same time building a database for
use in its long-term evaluation. Proposing that ATP's experience might be of
benefit to SBIR, Dr. Kelley provided a description of its evaluation methods:
· Keeping a database on projects and applicants "from the very beginning"
allows ATP to track the characteristics of the applicants, projects, and
types of technologies it funds.
· Progress on individual awards is charted in two ways:
Participants are required to fill out documents for ATP's internal use;
agreeing to provide information and to be studied is, she said, "in
some sense the quid pro quo for being an awardee of the ATP." These
documents help ATP staff gain knowledge about both the effective-
ness of the program and the issues that arise in conducting it that may
contribute to the quality of program management.
R&D activity is monitored by ATP project managers with the help of
staff visits to participating companies and technical reports that
awarders are required to provide on a quarterly and annual basis.
Studies are commissioned by ATP that may focus on a particular technol-
ogy area, a particular industry, or even particular firms, and that may
involve projections of the long-term benefits to be expected from certain
types of awards.
Methodological research questions are explored in the field of metrics,
addressing the question: "What do we need to know to be able to deter-
mine whether or not a project is successful?"
· Projects are assessed for their ability to meet the goal of providing broad
returns to the economy at large. Social benefits are not studied to the
exclusion of private returns, however, because the latter "are, after all,
part of the social return."
Because ATP's funding did not reach its present order of magnitude until 1994,
and because most of the awards have been to multiyear projects, Dr. Kelley noted,
very few projects have been completed. The program must be considered as still
at a very early point in the process.
Turning to data analyzed so far, Dr. Kelley said that 3,000 proposals from a
wide area of technological activities have been submitted over the life of the
program. These proposals have requested, in total, far more money than has been
available in ATP's budget, and therefore only a small percentage has been
awarded. Whether the proposals selected have been efforts undertaken by indi
OCR for page 96
96
THE SMALL BUSINESS INNOVATION AND RESEARCH PROGRAM
vidual companies or collaborative joint ventures involving groups of companies,
all have involved some degree of cost-sharing on the part of recipients. And
although ATP, far from targeting small firms, has large and small firms compet-
ing for awards, more than half of the projects funded under the program to date
have been to small firms or to consortia that had small firms as their leaders.
THE SBIR EXPERIENCE AT NIH
Herbert Kreitman
National Institutes of Health
Dr. Kreitman, the coordinator of the SBIR and the Small Business Technol-
ogy Transfer (STTR) programs for the U.S. Public Health Service and NIH,
opened by noting that NIH's budget has grown significantly over the past few
years to $13.6 billion in 1998. The Clinton administration has proposed that it
surpass $20 billion in 2003. In the current year, NIH SBIR and STTR set-asides
total $281 million, keeping the agency in second place among all SBIR agencies
behind the DoD.
Whether defined as sales within the federal government or to the private
sector, commercialization receives less attention from NIH than from most other
SBIR-participant agencies. Although commercialization is among NIH's review
criteria, more than 95 percent of the agency's SBIR awards are made as grants,
for which review criteria are weighted differently than when awarding contracts.
Furthermore, Mr. Kreitman suggested that the metric for commercialization be
expanded to include published papers describing the results of SBIR projects and
the use of SBIR technology in other research projects. Despite all this, he noted
that in the most recent U.S. Government Accounting Office (GAO) report on
SBIR commercialization, which dates to 1992, NIH was "held as the standard for
commercialization."
Describing other characteristics of the NIH SBIR program, he said that there
is no topic number on the face page of the agency's application form. It is the
investigator-initiated research project that NIH is most interested in, and any
project that fits its mission of biological and behavioral research is welcome. The
agency has a Fast Track program that is probably somewhat more flexible than
the DoD pilot. NIH's Fast Track, while it encourages matching in the amount of
a Phase II grant, does not require matching at any specified level and accepts it in
the form of resources as well as cash. NIH's Fast Track program also requires a
product development plan, which is not a requirement of its standard Phase I or
Phase II applications.
Mr. Kreitman concluded by noting that, although Congress has treated NIH
generously overall, it has placed restrictions on the agency's research and man-
agement support budget line. NIH believes it could do a much better job of
nurturing the small business community if it had additional resources for out
OCR for page 97
PANEL V
97
reach. Even though the small business community must be represented on the
agency's SBIR and STTR peer-review panels, the majority of reviewers come
from academia. "So, yes, there is grantsmanship. You have to, as best you can,
get into the heads of the reviewers," he said, adding: "We are perfectly willing to
provide such nurturing, but we need some resources to do that."
THE FAST TRACK PILOT
Jon Baron
U.S. Department of Defense
Mr. Baron specified at the outset that the central thrust of reforms made in
the DoD's SBIR program over the past two years has been to increase the
program's success in converting the research it funds into viable products that
would make a major difference to U.S. military capabilities. The reforms, of
which Fast Track is one, grew out of an assessment done in 1995 by Dr. Davis's
office within the Office of the Director of Defense Research and Evaluation
(DDR&E). Dr. Flamm, who headed DoD's dual-use technology office at the
time, and Dr. Robyn of the White House National Economic Council were
involved as well.
A process action team comprising representatives of various DoD agencies-
including the program managers for SBIR from the Army, Navy, Air Force, and
defense agencies developed the actual reforms. Although sales of products from
SBIR grantees to customers outside the department economy were also goals of
the reforms, Mr. Baron emphasized that the reforms' major objective was to ben-
efit the DoD itself in a significant way.
Overview of DoD's SBIR Program
The funding level for SBIR in DoD during FY 1997 was $547 million. This
represents not only half of the government' s entire SBIR program, but also one-
fourth of all federal R&D funds that go to small business. The Pentagon's SBIR
program is administered by eight component programs within overall policy guid-
ance from the Office of the Secretary of Defense. A 1997 GAO report focusing
specifically on DoD's SBIR program was "quite positive," Mr. Baron said, in line
with the consistently favorable reviews accorded the SBIR program as a whole in
a number of independent studies.
In addition, a study funded by Dr. Davis's office at DDR&E found that
Phase II awards under DoD's program between 1984 and 1992 yielded average
product sales in Phase III of $760,000. That study virtually replicated a 1992
GAO study of commercialization, which had placed average product sales in
Phase III at $285,000; the substantial increase presumably stems from the fact
OCR for page 98
98
THE SMALL BUSINESS INNOVATION AND RESEARCH PROGRAM
that the technologies considered have had more time to mature into actual
products.
As an example of the program's success, Mr. Baron named Viasat, an SBIR
start-up company funded by both the Air Force and the Navy that developed a
technology used in military communications satellites. Before the implementa-
tion of the technology, each user of a military communications satellite had to set
up a dedicated channel. This would tie up the satellite for a certain period of time
during which no others could use it. By obviating the necessity of a user's tying
up a satellite other than when it is actually communicating, Viasat's technology
has effectively enabled military communications satellites to handle up to ten
times as many users in the course of a day. Sales to DoD now stand at about $90
million, and the technology has begun generating significant sales to the private
sector for commercial communications satellites. The commercial market prom-
ises to exceed $100 million; orders from AT&T and others have already surpassed
$7 million.
DoD's Fast Track Pilot
In 1995 the DoD process action team made a careful review of the
department's SBIR program, looked at previous studies by GAO and others, and
spoke with a number of companies that had participated in the program in par-
ticular, but not exclusively, companies like Dr. Banucci's that had become "huge"
successes. Based on that review, the team came to a conclusion very similar to
that of both the DDR&E and GAO the studies: Although the program was very
successful, the success was concentrated, with the top 1.5 percent of Phase II
award winners accounting for over 50 percent of that success as measured in
Phase III sales.
In particular, the team concluded that many of the companies taking part in
the SBIR program are very strong in R&D capability. However, they are weak in
the capabilities that would translate their R&D into real products that could be
sold to DoD. Companies that had won ten or more SBIR awards in the period
1984-1992 had on average about 40 percent of the sales resulting to both DoD
and the private sector. "These are companies that write good proposals [and] do
good research and development," Mr. Baron observed. "Their research and devel-
opment has had some benefit to the department, but not the same significant
benefit as if [it had been] converted into actual products that were sold to DoD in
significant numbers and incorporated into defense systems."
Among the reforms put forward by the team to address this problem was the
SBIR Fast Track pilot project. Under Fast Track, SBIR projects that attract "a
little bit" of matching cash from outside investors toward the end of Phase I gain
a significantly higher chance of obtaining a Phase II award; they also receive
expedited processing and interim funding that, together, amount to continuous
funding between Phase I and Phase II. As explained by Mr. Baron, the rationale
OCR for page 99
PANEL V
99
for this policy is that if a company attracts matching money whether from a
venture-capital firm, defense contractor, or other company, or by meshing with
an acquisition program within DoD this sends an important signal. "This says
more about the size of the market for its technology and about the company's
ability to bring that technology to market," said Mr. Baron, "than anything the
company could write down on a piece of paper in a proposal."
Although it is still in the pilot stage, Mr. Baron said that results so far show
that, in general, Fast Track has been meeting its intended goals. Last year, with
the help of DDR&E and the managers of the component SBIR programs at DoD,
the pilot was streamlined and some administrative problems were corrected,
whereupon it was extended for two years. Scheduled for next year is an indepen-
dent, systematic evaluation of Fast Track that is designed to determine whether
the favorable early results have been sustained and whether the improvements
have accomplished their objectives. Anecdotal evidence so far indicates that Fast
Track has provided an effective means for companies to leverage their SBIR
awards to obtain outside funding. This is because, as Mr. Baron explained, Fast
Track has allowed them to entice potential partners with the prospect of their
investments' being matched at ratios of between one-to-one and four-to-one by
the Defense Department.
Returning to the DDR&E study, Mr. Baron noted that private-sector invest-
ment in SBIR projects seems to correlate strongly with whether the awardee will
have success in generating sales of new products. According to the study, SBIR
awarders that had failed to attract any private-sector investment had about a 2
percent chance of achieving sales a few years down the line. For those that had
attracted more than $1 million in outside investment, that chance rose to about 40
percent. This holds for a subset of SBIR awarders in which sales are defined
solely as sales to the DoD. If an SBIR project had no private-sector investment,
it had about a 3 percent chance of achieving sales to DoD or to DoD prime con-
tractors in Phase III, a figure that went as high as 30 percent for awarders attract-
ing outside capital. This evidence, he said, very strongly suggests that, contrary
to claims that conflict exists between agency missions and the goal of private-
sector commercialization, the two objectives are in fact complementary. The
central challenge is getting the R&D converted into saleable products, regardless
of who the customer is.
Other DoD SBIR Reforms
Mr. Baron concluded what he characterized as a "very preliminary" report
by enumerating other changes enacted as a result of proposals by DoD's SBIR
process action team:
Phase II declaration: A company submitting an SBIR proposal must now
list all Phase II awards it has won, as well as declare the sales revenues stemming
OCR for page 100
100
THE SMALL BUSINESS INNOVATION AND RESEARCH PROGRAM
from those prior Phase II awards and the amount of non-SBIR money it has raised
to match them. The Ballistic Missile Defense Organization and some other orga-
nizations use this practice in their evaluation processes; at this point, it is unclear
what effect this has had.
Streamlining: DoD has made an effort to streamline the processing of SBIRs
across the department, and thanks to the work of the component programs there
has been a 30 percent reduction in funding delays. Because time to market is
critical in most high-technology industries, the year's delay between Phase I and
Phase II that the department experienced in the past had biased the program
against success in the private sector.
Assessment: Prospective SBIR proposers can now talk with the DoD scien-
tists and engineers who author the program topics before submission. This inno-
vation, which originated in the Air Force and was also recommended by DDR&E,
allows companies that have never done business with the department to get a
much better sense of its needs before they submit a proposal.
DISCUSSANT
Charles Rowe
House Committee on Small Business
Claiming he lacks the qualifications to comment on "a selection and peer-
review process that's placed mostly in the hands of people with Ph.D.s,"
Mr. Rowe indicated that his remarks would focus on the results of the SBIR
program. These results, as appraised by the National Academy of Sciences, GAO,
or the Small Business Administration, have shown the program to have been
effective over the years. "Although the vast majority of federal R&D funding
goes to universities, large businesses, and federal laboratories, we see that the
vast majority of patent applications are coming out of the small business end of
the pipeline," he stated. "That is only to be expected from a program that is
fundamentally based on competition, merit, and commercialization. There is not
much reason to patent something that you are not planning to sell."
Distribution of Awards
What has been termed a "lopsided distribution of awards" is the largest single
problem with the selection process that has been brought to the committee's
attention over the years. Calling this phenomenon "both a boon and a curse to the
program," Mr. Rowe signaled his approval of the fact that awards have been
concentrated in certain geographical areas. Pointing out that, according to the
study presented by Mr. Tibbetts, award winners are usually located in "places
where the money is," he argued that the presence of money aids Phase III com-
mercialization. Calling the SBIR program competitive and the proposals it attracts
OCR for page 101
PANEL V
101
excellent, he said that the committee has yet to see a single concrete example of
an award that was unworthy of funding, while a desire to find money to fund
more of the proposals has been evident.
The Phase I and Phase II Funding Gap
A second and far more concrete complaint coming out of the selection pro-
cess involves the gap between Phase I and Phase II funding that DoD's Fast
Track program is designed to remove. Mr. Rowe expressed his pleasure at noting
that NIH has put a version of Fast Track in place as well.
Although there may be problems with some versions of the selection process
at the 11 agencies that participate in SBIR, Congress in creating the program had
very deliberately kept its fingers out of the management of the selection process.
Leaving it to the agencies themselves creates a spirit of competition among them
that will be far more productive in the long run than micromanagement from
Capitol Hill.
DISCUSSION
Ann Eskesen of the Innovative Development Institute stressed the impor-
tance of Dr. Nelson' s observation at a previous session that the SBIR program' s
framework allowed him the latitude to design his own program. This allowed
him to get to know the companies participating in it extremely well. Similarly,
she underscored Mr. Kreitman's point that an agency running a program on the
scale of that at NIH needs the resources to stay in contact with the companies
involved. "There very clearly is a correlation between the effectiveness of par-
ticipation in the SBIR program and the amount of interaction that occurs," she
stated, "whether it occurs within the agencies, to the companies, or within those
companies and the state support organizations."
Looking forward to SBIR's reauthorization, Ms. Eskesen advocated moving
beyond the outreach effort that she views as having been the program's emphasis
since its creation in 1982. Focus on getting the small firms to participate in the
program has, to date, overshadowed concern for Phase III, during which it was
simply "assumed that something was going to happen." Although "for the really
good companies it did happen," the lesson of the past few years is that some
companies need support and access to resources rather than just the incentive to
be involved. Because SBIR is now a mature program, and because the environ-
ment in which it is functioning is fundamentally different from that in which it
was created, the needs of awardee companies merit consideration as the reautho-
rization approaches. Referring to the difficulty many companies have in bridging
the "funding gap" between R&D and demonstration, she said that "setting stan-
dards and just pushing the bar up higher and higher works for some companies
but is not working for all."
OCR for page 102
102
THE SMALL BUSINESS INNOVATION AND RESEARCH PROGRAM
Ms. Eskesen then turned to the general question of how the integrity of the
SBIR program is to be maintained in light of its sheer diversity of involvement.
Declaring that one size does not fit all in the case either of the agencies or of the
companies involved, she urged an examination of how micro-management is to
be avoided and individual program managers left with the freedom to experi-
ment. As the program has become institutionalized with age, and as its rising
funding level has brought increased visibility, this freedom has diminished. "With
all due respect to the present set of program managers," she stated, "a lot of your
predecessors made career-threatening decisions. They got away with it because
the SBIR program was small and somewhat in the corner, and most people did
not even know it was there."
Stressing the value of interaction, Ms. Eskesen observed that managers in
some agencies have little or no contact with the companies in their awards portfo-
lios. Technical officers she observed at NASA flight centers, for example, "would
not have known their companies if they walked through the door." They simply
had no resources that would have allowed them to get to know the firms.
Although the SBIR program works in its present form, improvements are neces-
sary if it is to fulfill its potential, Ms. Eskesen stated. She invited the panelists to
name one or two changes that might make a difference to SBIR personnel who
interact with companies and make the operational decisions.
Representative terms from entire chapter:
fast track
