Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Pane! V Improving Assessment and Selection INTRODUCTION Dan Roos Massachusetts Institute of Technology Dr. Roos opened the session with the announcement that Herbert Kreitman would substitute for Wendy Baldwin as the representative of the National Insti- tutes of Health (NIH) on the panel. He then introduced Maryellen Kelley, serving as a senior economist for the Advanced Technology Program (ATP) at the National Institute of Standards and Technology while on leave from her post as associate professor at the Heinz School of Public Policy and Management at Carnegie Mellon University. Dr. Kelley discussed the lessons emerging from ATP from two perspectives: (1) to what extent small businesses are involved in the program, and (2) how ATP compares with the SBIR program from the view- point of assessment, selection, and objectives. LESSONS FROM THE ADVANCED TECHNOLOGY PROGRAM Maryellen R. Kelley NIST and Carnegie Mellon University Differences between ATP and SBIR Beginning with a description of ATP's operation, Dr. Kelley outlined fea- tures of the program that make it unique in the federal government's technology 93
94 THE SMALL BUSINESS INNOVATION AND RESEARCH PROGRAM policy portfolio. ATP is aimed at directly supporting R&D activities that have a commercial orientation from their inception, unlike SBIR which pursues the goal of commercialization through R&D that supports the mission of the SBIR- member agency making the award. Like SBIR, as well as most government agencies that are engaged in R&D, ATP funds activities that are undertaken by private companies; in ATP's case, a cost-sharing requirement is designed to leverage private R&D investment with the goal of achieving public benefits. ATP is intended to support "enabling" technologies, which are characterized by both high technical risk and the potential for substantial, widespread commer- cial application. Technologies considered enabling can be: . · path-breaking those that promise to induce radical innovations; · infrastructural those that support R&D, production, or businesses in entire industries; or multi-use those that either become platforms for multiple paths to further technology development or are "generic" in that they have many applications. Rather than advancing a specific agency mission, the program's endeavors are "industry driven." An annual general competition is open to proposals from all fields of technology; in addition, there are numerous competitions each year devoted to preselected areas of technology focus that have been identified by ATP program managers in cooperation with industry. Evaluating ATP Study of the impact of the program, which was inaugurated in 1990, has been limited to date because of the long-term nature of the projects it funds. In the early stages, benefits that become apparent are largely in the generation of knowl- edge knowledge that may, to some extent, also be of use to non-ATP firms and for technologies other than those directly targeted by an award. In the long term, knowledge spillovers are expected, as are other types of benefits that do not accrue exclusively to those involved directly in the innovation process. In the case of infrastructure-enabling technologies, other firms may learn about the technolo- gies developed within ATP and build upon them. Also envisioned are market spillovers: Knowledge that reaches beyond the ATP-awardee firms may allow other firms to build cheaper and better products, eventually benefiting the consumer. "These are the ways in which the larger benefits of the program are realized in the economy," explained Dr. Kelley. "That, in some sense, is the major justi- fication for this type of effort: that it will have benefits beyond simply the private returns to the individual firms."
PANEL V . . 95 Implications of ATP Evaluation for SBIR ATP sees monitoring and evaluation as integral to an ongoing process of refining the program so that spillover benefits can be captured more effectively. Tracking projects as they progress, ATP' s management is able to consider impli- cations for the program's strategy while at the same time building a database for use in its long-term evaluation. Proposing that ATP's experience might be of benefit to SBIR, Dr. Kelley provided a description of its evaluation methods: · Keeping a database on projects and applicants "from the very beginning" allows ATP to track the characteristics of the applicants, projects, and types of technologies it funds. · Progress on individual awards is charted in two ways: Participants are required to fill out documents for ATP's internal use; agreeing to provide information and to be studied is, she said, "in some sense the quid pro quo for being an awardee of the ATP." These documents help ATP staff gain knowledge about both the effective- ness of the program and the issues that arise in conducting it that may contribute to the quality of program management. R&D activity is monitored by ATP project managers with the help of staff visits to participating companies and technical reports that awarders are required to provide on a quarterly and annual basis. Studies are commissioned by ATP that may focus on a particular technol- ogy area, a particular industry, or even particular firms, and that may involve projections of the long-term benefits to be expected from certain types of awards. Methodological research questions are explored in the field of metrics, addressing the question: "What do we need to know to be able to deter- mine whether or not a project is successful?" · Projects are assessed for their ability to meet the goal of providing broad returns to the economy at large. Social benefits are not studied to the exclusion of private returns, however, because the latter "are, after all, part of the social return." Because ATP's funding did not reach its present order of magnitude until 1994, and because most of the awards have been to multiyear projects, Dr. Kelley noted, very few projects have been completed. The program must be considered as still at a very early point in the process. Turning to data analyzed so far, Dr. Kelley said that 3,000 proposals from a wide area of technological activities have been submitted over the life of the program. These proposals have requested, in total, far more money than has been available in ATP's budget, and therefore only a small percentage has been awarded. Whether the proposals selected have been efforts undertaken by indi
96 THE SMALL BUSINESS INNOVATION AND RESEARCH PROGRAM vidual companies or collaborative joint ventures involving groups of companies, all have involved some degree of cost-sharing on the part of recipients. And although ATP, far from targeting small firms, has large and small firms compet- ing for awards, more than half of the projects funded under the program to date have been to small firms or to consortia that had small firms as their leaders. THE SBIR EXPERIENCE AT NIH Herbert Kreitman National Institutes of Health Dr. Kreitman, the coordinator of the SBIR and the Small Business Technol- ogy Transfer (STTR) programs for the U.S. Public Health Service and NIH, opened by noting that NIH's budget has grown significantly over the past few years to $13.6 billion in 1998. The Clinton administration has proposed that it surpass $20 billion in 2003. In the current year, NIH SBIR and STTR set-asides total $281 million, keeping the agency in second place among all SBIR agencies behind the DoD. Whether defined as sales within the federal government or to the private sector, commercialization receives less attention from NIH than from most other SBIR-participant agencies. Although commercialization is among NIH's review criteria, more than 95 percent of the agency's SBIR awards are made as grants, for which review criteria are weighted differently than when awarding contracts. Furthermore, Mr. Kreitman suggested that the metric for commercialization be expanded to include published papers describing the results of SBIR projects and the use of SBIR technology in other research projects. Despite all this, he noted that in the most recent U.S. Government Accounting Office (GAO) report on SBIR commercialization, which dates to 1992, NIH was "held as the standard for commercialization." Describing other characteristics of the NIH SBIR program, he said that there is no topic number on the face page of the agency's application form. It is the investigator-initiated research project that NIH is most interested in, and any project that fits its mission of biological and behavioral research is welcome. The agency has a Fast Track program that is probably somewhat more flexible than the DoD pilot. NIH's Fast Track, while it encourages matching in the amount of a Phase II grant, does not require matching at any specified level and accepts it in the form of resources as well as cash. NIH's Fast Track program also requires a product development plan, which is not a requirement of its standard Phase I or Phase II applications. Mr. Kreitman concluded by noting that, although Congress has treated NIH generously overall, it has placed restrictions on the agency's research and man- agement support budget line. NIH believes it could do a much better job of nurturing the small business community if it had additional resources for out
PANEL V 97 reach. Even though the small business community must be represented on the agency's SBIR and STTR peer-review panels, the majority of reviewers come from academia. "So, yes, there is grantsmanship. You have to, as best you can, get into the heads of the reviewers," he said, adding: "We are perfectly willing to provide such nurturing, but we need some resources to do that." THE FAST TRACK PILOT Jon Baron U.S. Department of Defense Mr. Baron specified at the outset that the central thrust of reforms made in the DoD's SBIR program over the past two years has been to increase the program's success in converting the research it funds into viable products that would make a major difference to U.S. military capabilities. The reforms, of which Fast Track is one, grew out of an assessment done in 1995 by Dr. Davis's office within the Office of the Director of Defense Research and Evaluation (DDR&E). Dr. Flamm, who headed DoD's dual-use technology office at the time, and Dr. Robyn of the White House National Economic Council were involved as well. A process action team comprising representatives of various DoD agencies- including the program managers for SBIR from the Army, Navy, Air Force, and defense agencies developed the actual reforms. Although sales of products from SBIR grantees to customers outside the department economy were also goals of the reforms, Mr. Baron emphasized that the reforms' major objective was to ben- efit the DoD itself in a significant way. Overview of DoD's SBIR Program The funding level for SBIR in DoD during FY 1997 was $547 million. This represents not only half of the government' s entire SBIR program, but also one- fourth of all federal R&D funds that go to small business. The Pentagon's SBIR program is administered by eight component programs within overall policy guid- ance from the Office of the Secretary of Defense. A 1997 GAO report focusing specifically on DoD's SBIR program was "quite positive," Mr. Baron said, in line with the consistently favorable reviews accorded the SBIR program as a whole in a number of independent studies. In addition, a study funded by Dr. Davis's office at DDR&E found that Phase II awards under DoD's program between 1984 and 1992 yielded average product sales in Phase III of $760,000. That study virtually replicated a 1992 GAO study of commercialization, which had placed average product sales in Phase III at $285,000; the substantial increase presumably stems from the fact
98 THE SMALL BUSINESS INNOVATION AND RESEARCH PROGRAM that the technologies considered have had more time to mature into actual products. As an example of the program's success, Mr. Baron named Viasat, an SBIR start-up company funded by both the Air Force and the Navy that developed a technology used in military communications satellites. Before the implementa- tion of the technology, each user of a military communications satellite had to set up a dedicated channel. This would tie up the satellite for a certain period of time during which no others could use it. By obviating the necessity of a user's tying up a satellite other than when it is actually communicating, Viasat's technology has effectively enabled military communications satellites to handle up to ten times as many users in the course of a day. Sales to DoD now stand at about $90 million, and the technology has begun generating significant sales to the private sector for commercial communications satellites. The commercial market prom- ises to exceed $100 million; orders from AT&T and others have already surpassed $7 million. DoD's Fast Track Pilot In 1995 the DoD process action team made a careful review of the department's SBIR program, looked at previous studies by GAO and others, and spoke with a number of companies that had participated in the program in par- ticular, but not exclusively, companies like Dr. Banucci's that had become "huge" successes. Based on that review, the team came to a conclusion very similar to that of both the DDR&E and GAO the studies: Although the program was very successful, the success was concentrated, with the top 1.5 percent of Phase II award winners accounting for over 50 percent of that success as measured in Phase III sales. In particular, the team concluded that many of the companies taking part in the SBIR program are very strong in R&D capability. However, they are weak in the capabilities that would translate their R&D into real products that could be sold to DoD. Companies that had won ten or more SBIR awards in the period 1984-1992 had on average about 40 percent of the sales resulting to both DoD and the private sector. "These are companies that write good proposals [and] do good research and development," Mr. Baron observed. "Their research and devel- opment has had some benefit to the department, but not the same significant benefit as if [it had been] converted into actual products that were sold to DoD in significant numbers and incorporated into defense systems." Among the reforms put forward by the team to address this problem was the SBIR Fast Track pilot project. Under Fast Track, SBIR projects that attract "a little bit" of matching cash from outside investors toward the end of Phase I gain a significantly higher chance of obtaining a Phase II award; they also receive expedited processing and interim funding that, together, amount to continuous funding between Phase I and Phase II. As explained by Mr. Baron, the rationale
PANEL V 99 for this policy is that if a company attracts matching money whether from a venture-capital firm, defense contractor, or other company, or by meshing with an acquisition program within DoD this sends an important signal. "This says more about the size of the market for its technology and about the company's ability to bring that technology to market," said Mr. Baron, "than anything the company could write down on a piece of paper in a proposal." Although it is still in the pilot stage, Mr. Baron said that results so far show that, in general, Fast Track has been meeting its intended goals. Last year, with the help of DDR&E and the managers of the component SBIR programs at DoD, the pilot was streamlined and some administrative problems were corrected, whereupon it was extended for two years. Scheduled for next year is an indepen- dent, systematic evaluation of Fast Track that is designed to determine whether the favorable early results have been sustained and whether the improvements have accomplished their objectives. Anecdotal evidence so far indicates that Fast Track has provided an effective means for companies to leverage their SBIR awards to obtain outside funding. This is because, as Mr. Baron explained, Fast Track has allowed them to entice potential partners with the prospect of their investments' being matched at ratios of between one-to-one and four-to-one by the Defense Department. Returning to the DDR&E study, Mr. Baron noted that private-sector invest- ment in SBIR projects seems to correlate strongly with whether the awardee will have success in generating sales of new products. According to the study, SBIR awarders that had failed to attract any private-sector investment had about a 2 percent chance of achieving sales a few years down the line. For those that had attracted more than $1 million in outside investment, that chance rose to about 40 percent. This holds for a subset of SBIR awarders in which sales are defined solely as sales to the DoD. If an SBIR project had no private-sector investment, it had about a 3 percent chance of achieving sales to DoD or to DoD prime con- tractors in Phase III, a figure that went as high as 30 percent for awarders attract- ing outside capital. This evidence, he said, very strongly suggests that, contrary to claims that conflict exists between agency missions and the goal of private- sector commercialization, the two objectives are in fact complementary. The central challenge is getting the R&D converted into saleable products, regardless of who the customer is. Other DoD SBIR Reforms Mr. Baron concluded what he characterized as a "very preliminary" report by enumerating other changes enacted as a result of proposals by DoD's SBIR process action team: Phase II declaration: A company submitting an SBIR proposal must now list all Phase II awards it has won, as well as declare the sales revenues stemming
100 THE SMALL BUSINESS INNOVATION AND RESEARCH PROGRAM from those prior Phase II awards and the amount of non-SBIR money it has raised to match them. The Ballistic Missile Defense Organization and some other orga- nizations use this practice in their evaluation processes; at this point, it is unclear what effect this has had. Streamlining: DoD has made an effort to streamline the processing of SBIRs across the department, and thanks to the work of the component programs there has been a 30 percent reduction in funding delays. Because time to market is critical in most high-technology industries, the year's delay between Phase I and Phase II that the department experienced in the past had biased the program against success in the private sector. Assessment: Prospective SBIR proposers can now talk with the DoD scien- tists and engineers who author the program topics before submission. This inno- vation, which originated in the Air Force and was also recommended by DDR&E, allows companies that have never done business with the department to get a much better sense of its needs before they submit a proposal. DISCUSSANT Charles Rowe House Committee on Small Business Claiming he lacks the qualifications to comment on "a selection and peer- review process that's placed mostly in the hands of people with Ph.D.s," Mr. Rowe indicated that his remarks would focus on the results of the SBIR program. These results, as appraised by the National Academy of Sciences, GAO, or the Small Business Administration, have shown the program to have been effective over the years. "Although the vast majority of federal R&D funding goes to universities, large businesses, and federal laboratories, we see that the vast majority of patent applications are coming out of the small business end of the pipeline," he stated. "That is only to be expected from a program that is fundamentally based on competition, merit, and commercialization. There is not much reason to patent something that you are not planning to sell." Distribution of Awards What has been termed a "lopsided distribution of awards" is the largest single problem with the selection process that has been brought to the committee's attention over the years. Calling this phenomenon "both a boon and a curse to the program," Mr. Rowe signaled his approval of the fact that awards have been concentrated in certain geographical areas. Pointing out that, according to the study presented by Mr. Tibbetts, award winners are usually located in "places where the money is," he argued that the presence of money aids Phase III com- mercialization. Calling the SBIR program competitive and the proposals it attracts
PANEL V 101 excellent, he said that the committee has yet to see a single concrete example of an award that was unworthy of funding, while a desire to find money to fund more of the proposals has been evident. The Phase I and Phase II Funding Gap A second and far more concrete complaint coming out of the selection pro- cess involves the gap between Phase I and Phase II funding that DoD's Fast Track program is designed to remove. Mr. Rowe expressed his pleasure at noting that NIH has put a version of Fast Track in place as well. Although there may be problems with some versions of the selection process at the 11 agencies that participate in SBIR, Congress in creating the program had very deliberately kept its fingers out of the management of the selection process. Leaving it to the agencies themselves creates a spirit of competition among them that will be far more productive in the long run than micromanagement from Capitol Hill. DISCUSSION Ann Eskesen of the Innovative Development Institute stressed the impor- tance of Dr. Nelson' s observation at a previous session that the SBIR program' s framework allowed him the latitude to design his own program. This allowed him to get to know the companies participating in it extremely well. Similarly, she underscored Mr. Kreitman's point that an agency running a program on the scale of that at NIH needs the resources to stay in contact with the companies involved. "There very clearly is a correlation between the effectiveness of par- ticipation in the SBIR program and the amount of interaction that occurs," she stated, "whether it occurs within the agencies, to the companies, or within those companies and the state support organizations." Looking forward to SBIR's reauthorization, Ms. Eskesen advocated moving beyond the outreach effort that she views as having been the program's emphasis since its creation in 1982. Focus on getting the small firms to participate in the program has, to date, overshadowed concern for Phase III, during which it was simply "assumed that something was going to happen." Although "for the really good companies it did happen," the lesson of the past few years is that some companies need support and access to resources rather than just the incentive to be involved. Because SBIR is now a mature program, and because the environ- ment in which it is functioning is fundamentally different from that in which it was created, the needs of awardee companies merit consideration as the reautho- rization approaches. Referring to the difficulty many companies have in bridging the "funding gap" between R&D and demonstration, she said that "setting stan- dards and just pushing the bar up higher and higher works for some companies but is not working for all."
102 THE SMALL BUSINESS INNOVATION AND RESEARCH PROGRAM Ms. Eskesen then turned to the general question of how the integrity of the SBIR program is to be maintained in light of its sheer diversity of involvement. Declaring that one size does not fit all in the case either of the agencies or of the companies involved, she urged an examination of how micro-management is to be avoided and individual program managers left with the freedom to experi- ment. As the program has become institutionalized with age, and as its rising funding level has brought increased visibility, this freedom has diminished. "With all due respect to the present set of program managers," she stated, "a lot of your predecessors made career-threatening decisions. They got away with it because the SBIR program was small and somewhat in the corner, and most people did not even know it was there." Stressing the value of interaction, Ms. Eskesen observed that managers in some agencies have little or no contact with the companies in their awards portfo- lios. Technical officers she observed at NASA flight centers, for example, "would not have known their companies if they walked through the door." They simply had no resources that would have allowed them to get to know the firms. Although the SBIR program works in its present form, improvements are neces- sary if it is to fulfill its potential, Ms. Eskesen stated. She invited the panelists to name one or two changes that might make a difference to SBIR personnel who interact with companies and make the operational decisions.
