B. OVERVIEW OF THE PAPERS
Studies from the Assessment Program
A variety of evaluation methods, both qualitative and quantitative, are now being used to assess the effectiveness of ATP awards. The papers collected here are drawn from a growing body of studies focused on the ATP contribution and awardee performance. These studies have focused on a range of issues such as the ATP's effect on the innovation process, the commercial progress of the innovators, various spillover effects, and combinations of these effects, while others seek to improve the tools of evaluation needed to measure performance. Many of these studies are quite extensive (e.g., the study of the printed wiring board consortium). 121 In these cases, summary versions of the studies were prepared to illustrate the methodology and summarize the main points.
Assessing the ATP Contribution
The first paper, led by Professor Maryann Feldman of Johns Hopkins University, is particularly important in that it focuses on the ATP contribution to private-sector innovation. Using data from a survey of 1998 ATP applicants, the study finds that most of the non-winners did not proceed with any aspect of their proposed R&D project, and, of those that did, most did so on a smaller scale than initially proposed. This suggests that ATP funding is not simply displacing private capital.
Importantly, the program received high marks from its users. A substantial majority of the applicants surveyed by Feldman and Kelley widely considered ATP's application process to be fair and rational. Further, the survey finds that the projects and firms selected by ATP are more willing than those not selected to share their research findings with other firms and tend to be collaborative in new technical areas and form new R&D partnerships—findings consistent with ATP's goal of selecting projects with large spillover potential. The study also finds that the ATP award can create a “halo effect” for recipients, increasing the success of award recipients in attracting additional funding from other sources, an effect documented by several earlier researchers. 122 Feldman and Kelley conclude that
121 A. N. Link, ATP Early Stage Impacts of the Printed Wiring Board Research Joint Venture, Assessed at Project End, NIST GCR 97-722, November 1997.
122 See Silber & Associates, Survey of Advanced Technology Program 1990-1992 Awardees: Company Opinion about the ATP and its Early Effects, NIST GCR 97-707, February 1996; and Solomon Associates, Advanced Technology Program: An Assessment of Short-Term Impacts—First Competition Participants, February 1993. See also J. Lerner, ” ‘Public Venture Capital': Rationales and Evaluation,” in National Research Council, The Small Business Innovation Research Program: Challenges and Opportunities, Charles W. Wessner, editor, Washington, D.C.: National Academy Press, 1999, pp. 115-128.
the ATP is leveraging activities that have the potential to contribute to broad-based economic growth.
Accelerating Collaborative Activity
The architects of the ATP apparently anticipated the growing importance of collaborative activity to national technological prowess, and made the fostering of collaboration a prominent feature of the ATP. Some of the ATP evaluations therefore focus on the collaborative aspect of ATP awards. Professor Jeffrey Dyer of the Wharton School and, more recently, Brigham Young University, identifies factors that increase or decrease the likelihood of collaborative success from the perspective of participants in ATP-funded joint ventures in the automotive industry. His findings suggest that
the ATP is accelerating and improving the successful outcome of collaborative projects;
the ATP projects are taking on higher risk and longer-term research than collaborative endeavors without government involvement; and
the ATP is providing funding during critical stages, overcoming barriers to collaborating, increasing project stability, and causing collaborative projects to run more smoothly, albeit with some perceived loss of flexibility on the part of participants.
Several papers are case studies designed to measure various elements of performance of specific projects or collections of projects.
The earliest of these, by Professor Albert Link of the University of North Carolina-Greensboro, also addresses the value of collaboration. He focuses on the impact of an ATP-funded joint venture on the costs and timing of developing a suite of “leap-frog” technologies for the U.S. printed wiring board industry. Link finds major R&D efficiency gains from the project, estimating the resulting cost savings in the millions of dollars. He also finds that the competitive positions of U.S. producers improved substantially and attributes this to their strengthened technical capabilities. The resulting employment effects are reportedly positive and substantial. 123123 The head of the National Center for Manufacturing Sciences declared that the project was instrumental in turning around the declining wire board industry, contributing to the retention of approximately 200,000 jobs.
Medical Technology: Tissue Engineering
The study by researchers at the Research Triangle Institute's (RTI) Center for Economic Research develops an evaluation framework for assessing medical technologies to apply to seven ATP-funded tissue-engineering projects. These projects offer new medical treatments, mostly at lower costs. RTI computes three measures of benefit for each project: (1) social return on investment; (2) private return on investment; and (3) social return on public investment, or the return on ATP's investment based on the difference in social return with and without the ATP. They compute a composite social return across the seven projects in the billions of dollars (net present value). They conclude that expected social returns are much larger than private returns, primarily due to projected positive spillovers to patients treated with the new technologies. Also, they conclude that the ATP played a significant role in increasing expected returns on these projects by accelerating the R&D phase of the projects and improving the probability of technical success.
An Economic Model
In the fifth study, a more theoretical approach is taken by two economists at Resources for the Future, David Austin and Molly Macauley. This paper projects potential gains from ATP-supported technologies and in addition illustrates the contributions the ATP's evaluation program has made to the increasingly important field of technology impact assessment. The ATP has engaged leading economists and other evaluators to apply state-of-the-art evaluation techniques to assess the impact of the program. In doing this, it has also commissioned work that extends the state of the art of evaluating new technologies by testing new methods on ATP-funded projects. The work by Austin and Macauley may represent an advance in evaluation modeling capability, as well as adding to our knowledge of the potential impact of two ATP-funded projects in digital data storage. This study estimates that each of these projects, if successfully commercialized, would generate consumer benefits in excess of a billion dollars. It is important to note that the findings are dependent on successful commercialization of the technologies, which is not assured.
The final paper, prepared by Rosalie Ruegg, provides an aggregate analysis of the first fifty completed ATP projects drawn from small case studies compiled by a number of analysts. This overview describes the results of a substantial group of ATP-funded projects, several years after completion. The fifty projects are rated in terms of the creation of new technical knowledge, dissemination of new knowledge to others, and direct use of the new knowledge by the innovators to
accelerate commercial use of the technology—three dimensions of performance that figure prominently in the long-run success of the ATP. As one would expect for high-risk R&D, many of the projects are rated as weak performers (26 percent); most of the projects (58 percent) fall somewhere in the middle. Yet expected net benefits from the strong performers (16 percent) are more than enough to yield a robust performance for the group of fifty taken as a whole. In the aggregate, the expected benefits of stronger performers would outweigh total program costs to date. Strictly speaking, this level of return is not the goal of the ATP. It does provide one measure of return on investment.
Conclusions and Caveats
These studies, performed by different researchers using different approaches, are generally positive in their findings, although the complexity and tentative nature of the more recent assessments must be kept in mind. Technologies now showing progress may nevertheless fail to deliver expected benefits. Individually and collectively, the analyses suggest that the ATP is making progress toward achieving its intended mission. In addition, the program's sustained evaluation effort is developing valuable tools for the assessment of U.S. technology policies.