National Academies Press: OpenBook

The Advanced Technology Program: Assessing Outcomes (2001)

Chapter: Overview of the Papers

« Previous: Background
Suggested Citation:"Overview of the Papers." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×

Page 59

B. OVERVIEW OF THE PAPERS

Studies from the Assessment Program

A variety of evaluation methods, both qualitative and quantitative, are now being used to assess the effectiveness of ATP awards. The papers collected here are drawn from a growing body of studies focused on the ATP contribution and awardee performance. These studies have focused on a range of issues such as the ATP's effect on the innovation process, the commercial progress of the innovators, various spillover effects, and combinations of these effects, while others seek to improve the tools of evaluation needed to measure performance. Many of these studies are quite extensive (e.g., the study of the printed wiring board consortium). 121 In these cases, summary versions of the studies were prepared to illustrate the methodology and summarize the main points.

Assessing the ATP Contribution

The first paper, led by Professor Maryann Feldman of Johns Hopkins University, is particularly important in that it focuses on the ATP contribution to private-sector innovation. Using data from a survey of 1998 ATP applicants, the study finds that most of the non-winners did not proceed with any aspect of their proposed R&D project, and, of those that did, most did so on a smaller scale than initially proposed. This suggests that ATP funding is not simply displacing private capital.

Importantly, the program received high marks from its users. A substantial majority of the applicants surveyed by Feldman and Kelley widely considered ATP's application process to be fair and rational. Further, the survey finds that the projects and firms selected by ATP are more willing than those not selected to share their research findings with other firms and tend to be collaborative in new technical areas and form new R&D partnerships—findings consistent with ATP's goal of selecting projects with large spillover potential. The study also finds that the ATP award can create a “halo effect” for recipients, increasing the success of award recipients in attracting additional funding from other sources, an effect documented by several earlier researchers. 122 Feldman and Kelley conclude that

121 A. N. Link, ATP Early Stage Impacts of the Printed Wiring Board Research Joint Venture, Assessed at Project End, NIST GCR 97-722, November 1997.

122 See Silber & Associates, Survey of Advanced Technology Program 1990-1992 Awardees: Company Opinion about the ATP and its Early Effects, NIST GCR 97-707, February 1996; and Solomon Associates, Advanced Technology Program: An Assessment of Short-Term Impacts—First Competition Participants, February 1993. See also J. Lerner, ” ‘Public Venture Capital': Rationales and Evaluation,” in National Research Council, The Small Business Innovation Research Program: Challenges and Opportunities, Charles W. Wessner, editor, Washington, D.C.: National Academy Press, 1999, pp. 115-128.

Suggested Citation:"Overview of the Papers." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×

Page 60

the ATP is leveraging activities that have the potential to contribute to broad-based economic growth.

Accelerating Collaborative Activity

The architects of the ATP apparently anticipated the growing importance of collaborative activity to national technological prowess, and made the fostering of collaboration a prominent feature of the ATP. Some of the ATP evaluations therefore focus on the collaborative aspect of ATP awards. Professor Jeffrey Dyer of the Wharton School and, more recently, Brigham Young University, identifies factors that increase or decrease the likelihood of collaborative success from the perspective of participants in ATP-funded joint ventures in the automotive industry. His findings suggest that

  • the ATP is accelerating and improving the successful outcome of collaborative projects;

  • the ATP projects are taking on higher risk and longer-term research than collaborative endeavors without government involvement; and

  • the ATP is providing funding during critical stages, overcoming barriers to collaborating, increasing project stability, and causing collaborative projects to run more smoothly, albeit with some perceived loss of flexibility on the part of participants.

Case Studies

Several papers are case studies designed to measure various elements of performance of specific projects or collections of projects.

Wiring Boards

The earliest of these, by Professor Albert Link of the University of North Carolina-Greensboro, also addresses the value of collaboration. He focuses on the impact of an ATP-funded joint venture on the costs and timing of developing a suite of “leap-frog” technologies for the U.S. printed wiring board industry. Link finds major R&D efficiency gains from the project, estimating the resulting cost savings in the millions of dollars. He also finds that the competitive positions of U.S. producers improved substantially and attributes this to their strengthened technical capabilities. The resulting employment effects are reportedly positive and substantial. 123

123 The head of the National Center for Manufacturing Sciences declared that the project was instrumental in turning around the declining wire board industry, contributing to the retention of approximately 200,000 jobs.
Suggested Citation:"Overview of the Papers." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×

Page 61

Medical Technology: Tissue Engineering

The study by researchers at the Research Triangle Institute's (RTI) Center for Economic Research develops an evaluation framework for assessing medical technologies to apply to seven ATP-funded tissue-engineering projects. These projects offer new medical treatments, mostly at lower costs. RTI computes three measures of benefit for each project: (1) social return on investment; (2) private return on investment; and (3) social return on public investment, or the return on ATP's investment based on the difference in social return with and without the ATP. They compute a composite social return across the seven projects in the billions of dollars (net present value). They conclude that expected social returns are much larger than private returns, primarily due to projected positive spillovers to patients treated with the new technologies. Also, they conclude that the ATP played a significant role in increasing expected returns on these projects by accelerating the R&D phase of the projects and improving the probability of technical success.

An Economic Model

In the fifth study, a more theoretical approach is taken by two economists at Resources for the Future, David Austin and Molly Macauley. This paper projects potential gains from ATP-supported technologies and in addition illustrates the contributions the ATP's evaluation program has made to the increasingly important field of technology impact assessment. The ATP has engaged leading economists and other evaluators to apply state-of-the-art evaluation techniques to assess the impact of the program. In doing this, it has also commissioned work that extends the state of the art of evaluating new technologies by testing new methods on ATP-funded projects. The work by Austin and Macauley may represent an advance in evaluation modeling capability, as well as adding to our knowledge of the potential impact of two ATP-funded projects in digital data storage. This study estimates that each of these projects, if successfully commercialized, would generate consumer benefits in excess of a billion dollars. It is important to note that the findings are dependent on successful commercialization of the technologies, which is not assured.

Aggregate Analysis

The final paper, prepared by Rosalie Ruegg, provides an aggregate analysis of the first fifty completed ATP projects drawn from small case studies compiled by a number of analysts. This overview describes the results of a substantial group of ATP-funded projects, several years after completion. The fifty projects are rated in terms of the creation of new technical knowledge, dissemination of new knowledge to others, and direct use of the new knowledge by the innovators to

Suggested Citation:"Overview of the Papers." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×

Page 62

accelerate commercial use of the technology—three dimensions of performance that figure prominently in the long-run success of the ATP. As one would expect for high-risk R&D, many of the projects are rated as weak performers (26 percent); most of the projects (58 percent) fall somewhere in the middle. Yet expected net benefits from the strong performers (16 percent) are more than enough to yield a robust performance for the group of fifty taken as a whole. In the aggregate, the expected benefits of stronger performers would outweigh total program costs to date. Strictly speaking, this level of return is not the goal of the ATP. It does provide one measure of return on investment.

Conclusions and Caveats

These studies, performed by different researchers using different approaches, are generally positive in their findings, although the complexity and tentative nature of the more recent assessments must be kept in mind. Technologies now showing progress may nevertheless fail to deliver expected benefits. Individually and collectively, the analyses suggest that the ATP is making progress toward achieving its intended mission. In addition, the program's sustained evaluation effort is developing valuable tools for the assessment of U.S. technology policies.

Suggested Citation:"Overview of the Papers." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×
Page 59
Suggested Citation:"Overview of the Papers." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×
Page 60
Suggested Citation:"Overview of the Papers." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×
Page 61
Suggested Citation:"Overview of the Papers." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×
Page 62
Next: Summary of Symposium Proceedings »
The Advanced Technology Program: Assessing Outcomes Get This Book
×
Buy Paperback | $80.00 Buy Ebook | $64.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

This report examines the operations of the APT, reviews its extensive assessment program, and provides NRC Committee findings concerning the ATP’s operations and recommendations for potential improvements to the program. The report includes a summary of a major conference held in April 2000 as well as seven papers, including surveys of the industry participants or users of the ATP program, a summary of the results of fifty awards, detailed assessments of major joint ventures, and a description of the current selection process. It is the most comprehensive study to date of the program’s origins, operations, achievements, and assessment. Its conclusion: the program works.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!