National Academies Press: OpenBook

The Advanced Technology Program: Assessing Outcomes (2001)

Chapter: Taking a Step Back: An Early Results Overview of Fifty ATP Awards

« Previous: Perspectives on the Determinants of Success in ATP-sponsored R&D Joint Ventures: The Views of Participants
Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×

Page 259

Taking a Step Back: An Early Results Overview of Fifty ATP Awards

Rosalie Ruegg

Technology Impact Assessment (TIA) Consulting

SUMMARY

This report reviews and rates the performance of the first 50 projects that have completed their ATP-funded research, several years after completion. It develops and applies a four-star rating system to assess project progress toward achieving ATP's goals in light of the outputs of each of the completed projects.

Top Performers

For a group of top performers, it examines net benefits and their implications for the overall success to date of the ATP. Findings are that 16 percent of the completed projects scored in the top category in terms of overall effectiveness. The top performers tended to be strong in solving challenging technical problems, making the new technical knowledge available to others, and directly using that knowledge to accelerate commercial use of the technology—three dimensions of performance that figure prominently in achieving the long-run success of the ATP. Half of them received awards for their technical accomplishments from outside parties, and more than half of the single-company leaders received outside recognition for their business accomplishments. All of the top performers had collaborative relationships, and attracted private capital for their follow-on commercialization efforts. All of the single-company leaders in the group expanded their employment substantially. And, all of them have a very strong outlook for continued progress.

Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×

Page 260

Low Performers

Twenty-six percent of the 50 projects were in the bottom group of performers. Although most of these produced technical outputs such as patents or papers, none of them were award-winning; all showed little or no further direct progress toward commercialization; and the outlook for the future through direct action of the award recipients was poor or uncertain at best.

Middle Performers

Most of the 50 projects (58 percent) fell in the middle group of performers— they produced solid technical accomplishments but overall were not as strong as the top group. In some cases, these projects were strong technically but had not demonstrated robust commercial progress, and, in a few cases, they had produced a technology with commercial strength but had little else to show in the way of sharing knowledge with others—a public-interest goal of the program.

Portfolio Perspective

From a portfolio perspective, the results look strong for the ATP: the estimated net benefits attributed to the program from the top performers of the first 50 completed projects exceed the cost of the entire program to date, suggesting that the program is on track to produce a high return for the nation.

Caveat

These positive results are subject to an important caveat: The projects were assessed in two different groups in two different periods—38 of the projects were assessed two years ago and 12 more recently. Given that technology development and commercialization take time and are characterized by unexpected break-throughs and failures, future updates of these projects may alter the findings reported here.

SIGNIFICANCE OF THE FIRST 50 PROJECTS

From the moment the ATP funded its first group of 11 projects in the 1990 competition, program administrators, the Administration, Congress, technology policy makers, industry, and others in this country and abroad were keenly interested in the outcome. But technology development and commercialization are lengthy processes, and it takes time to produce results. Now, as the program completes its first decade of operation, there are a growing number of projects that have completed their ATP-funded research and moved into the post-project period. This group of completed projects makes it possible to measure project out

Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×

Page 261

puts, assess project progress and outlook, develop composite performance metrics, compile statistics, and analyze the results for implications about overall program success. A partial step was taken toward this goal with the publication of a report in 1999 on the first 38 completed projects. 1 This report takes the next step, by extending coverage to 50 projects and adding composite performance metrics. 2

PROJECT CHARACTERISTICS

Small Business Focus, with University Links

The majority (84 percent) of the first 50 completed projects are single-applicant projects, and most (70 percent) are led by small businesses. Although only 16 percent of the 50 projects are joint ventures, 84 percent of them had collaborative relationships. As shown in Table 1, nearly half the projects had close R&D ties with universities, and more than half the projects formed collaborative arrangements to pursue commercialization.

Technology Concentrations

By area of technology, the highest concentration (38 percent) is in Electronics/Computer Hardware/Communications, followed by Manufacturing (22 percent), Biotechnology and Advanced Materials/Chemicals (14 percent each), and Information Technology (12 percent). 3 These technology concentrations differ from those found in the larger ATP portfolio, where information technology is the largest single area, followed by Advanced Materials/Chemicals. Most of these were funded in ATP's General Competitions, whereas Focused Program Competitions gave rise to the majority of projects in the larger ATP portfolio.

Cost Sharing

As Table 2 shows, the ATP funded slightly more than half the $121.5 million of single-applicant total project research costs. 4 Joint ventures, though a minority of projects, received about 40 percent of total ATP funding. Together, the ATP

1 See William F. Long, Performance of Completed Projects: Status Report Number 1, NIST Special Publication 950-1, March 1999.

2 For the full report on the first 50 completed projects, upon which this paper is based see Rosalie T. Ruegg et al., Performance of 50 Completed Projects: Status Report Number 2, NIST Special Publication 950-2, forthcoming.

3 The ATP classifies its projects into these five technology areas according to where the major technical challenges lie. In fact, most projects cut across technology areas.

4 The ATP cost share is higher for this group than overall because a high percentage of the single-applicant projects were led by small businesses with very low overhead which is the portion of costs that must be covered by industry.
Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×

Page 262

TABLE 1 Collaborative Activity

Type of Collaboration

Number of Projects

Percentage

Collaborating on R&D with other companies or non-university organizations

21

42

Close R&D ties with universities

24

48

Collaborating on R&D with other companies or non-university organizations OR close R&D ties with universities

33

66

Collaborating on commercialization with other organizations

27

54

Collaborating in one or more of the above ways

42

84



and industry spent a total of $208 million on the 50 projects, with the ATP paying slightly less than half the total of project costs. The ATP spent an average of $1.5 million per single-applicant project and an average of $4.9 million per joint-venture project. Across the 50 projects, the average total cost (ATP + industry) per project was $4.2 million, and the median project length was three years.

STUDY APPROACH: 50 MINI-CASE STUDIES

At the core of the study from which this paper is drawn are 50 mini-case studies covering each of the completed projects. Each of these briefly tells the project story, recounting its goals and challenges, describing the innovators and their respective roles, and assessing progress to date and the future outlook.

Although the particulars vary for each project, certain types of data are systematically collected for all of them. Consistent with ATP's mission, the evaluation focuses on collecting data related to the following dimensions of performance.

  • Knowledge creation and dissemination, which is assessed using the following data: recognition by other organizations of a project's technical

TABLE 2 ATP Funding, Industry Cost Share and Total Costs of 50 Completed Projects

Type of Project

ATP Funding ($million)

Industry Cost Share ($millions)

Total Project Costs ($millions)

ATP Share of Costs (Percent)

Industry Share of Costs (Percent)

Single-Applicant Projects

64.5

57.0

121.5

53

47

Joint-Venture Projects

39.5

47.0

86.5

46

54

All Completed Projects

104.0

104.0

208.0

50

50

Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×

Page 263

accomplishments; numbers of patents filed and granted; citations of patents by others; publications and presentations; collaborative relationships; and knowledge embodied in and disseminated through new products and processes.

  • Commercialization progress , which is tracked in terms of attraction of additional capital for continued pursuit of project goals, including resources provided by collaborative partners; entering the market with products and services; employment changes at the small-companies leading projects, and other indicators of growth; awards by other organizations for business accomplishments of project leaders; and the analyst's assessment of the future outlook for the technology based on all the information collected. 5

  • Overall project effectiveness. The approach is to compile all of the data for all of the projects, and to compute aggregate statistics of interest across the set of 50, such as the total number of patents and the percentage of projects whose technologies have been commercialized. Composite scores of overall project performance are constructed by combining output data for knowledge creation, knowledge dissemination, and commercialization, and outlook. 6 The output data and the composite performance score allow one to see at a glance how a project has performed thus far in terms of the two paths by which ATP delivers benefits to the nation: (1) the indirect path which relies on knowledge transfer from the innovators to others who in turn may use the knowledge for economic benefit, and (2) the direct path by which the U.S. award recipient/innovators directly pursue commercialization of the newly developed technologies. 7

  • Data for the 50 projects were collected from ATP project records, telephone interviews with company representatives, interviews with ATP project managers, independent sources such as Hoovers Online Company and Industry Network, company web sites, the U.S. Patent Office, and in-depth project studies performed by other analysts.



    5 ATP has long reported certain of these data to meet requirements of the Government Performance and Results Act (GPRA) for project metrics, such as numbers of patents and publications, and number of technologies commercialized. The GPRA is a legislative framework for requiring federal agencies to set strategic goals, measure performance, and report on the degree to which goals were met. An overview of the GPRA is provided in Appendix 1 of the General Accounting Office Executive Guide, Effectively Implementing the Government Performance and Results Act, Report no. GGD-96-118, Washington, D.C.: Government Accounting Office, 1996.

    6 The methodology for developing consolidated project performance metrics was developed and applied to the first 50 completed projects by Rosalie T. Ruegg, TIA Consulting, and is described by her in greater depth in the full report on which this paper is based.

    7 See Rosalie T. Ruegg's presentation in the proceedings of this volume,
    Panel II, for a description and illustrations of the direct and indirect pathways of ATP-project impact.

    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×

    Page 264

    A Snap-Shot of Progress

    The cases provide a snap shot of progress as of several years after the end of the ATP-funded project. Although all of the cases were performed about the same length of time after the ATP funding was completed, they may capture different stages in technology development. Information technology projects, for example, usually move faster from research to commercialization than do advanced materials and chemical projects, and, hence, may be expected to be further along several years after the ATP-funded research project is completed.

    Another difference among the cases is the date they were performed. There are two groups: (1) 38 projects were assessed between 1997 and late 1998, and (2) 12 projects were assessed over 1999 and early 2000. 8 Since developments continue to unfold for most of these projects, the earlier set of 38 may have changed significantly since the data were collected. Indeed, the 12 may also have changed.

    Future studies may add mini-cases for additional completed projects to this group of 50, also performed several years after completion; and future studies may also update these studies by reviewing the projects' progress over time.

    PROJECT PERFORMANCE: KNOWLEDGE CREATIONAND DISSEMINATION

    Each of the 50 completed ATP projects targeted a number of specific technical-knowledge discovery goals designed to achieve a new or better way of doing things. 9 The knowledge created by each project is the source of its future economic benefit, both for the innovator and for others who acquire the knowledge. It is a good starting place for assessing completed projects.

    Knowledge gains by the 50 projects range from mathematical algorithms underlying new software tools, to the science of growing human tissue, to new techniques for fabricating high-temperature superconducting devices, to new chemical formulations. The diversity reflects the fact that all but two of the projects were funded in the ATP's General Competitions, which cast a wide net for ideas regardless of technology area. 10

    8 William F. Long collected the data for the first 38. Jonathan Tucker, Chris Hansen, Josh Rosenberg, Jon Dryfus, Benjamin Fletcher, Kathleen McTigue, Michael Walsh, Mariah Tanner, and Karen Seeh collected the data for the next 12, supplemented by data collected by Rosalie T. Ruegg.

    9 In establishing the ATP, Congress directed that it add to the nation's scientific and technical knowledge base.


    10 ATP also funded “Focused Program” competitions from 1994-1998 that funded certain areas in greater depth. Projects from these competitions will appear in larger numbers in subsequent studies of completed projects.

    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×

    Page 265

    TABLE 3 Outside Recognition of Technical Achievements of the First 50 Completed Projects

    Project Awardee

    Year

    Awarding Organization

    Award

    American Superconductor

    1996

    R&D Magazine

    One of the 100 most important innovations of the year.

    American Superconductor

    1996

    Industry Week magazine

    Technology of the Year award.

    Communication Intelligence #1

    1997

    Arthritis Foundation

    “Ease-of-Use Seal of Commendation” for the development of natural handwriting technology, for use by disabled people who have trouble with keyboard entry.

    DuPont

    1993

    Microwave & Rf magazine

    One of the Top Products of 1993, for high-temperature superconductivity component technology.

    Engineering Animation

    1994

    Computerworld magazine

    Smithsonian Award, for the use of information technology in the field of medicine.

    Engineering Animation

    1995

    Association of Medical Illustrators

    Association of Medical Illustrators Award of Excellence in Animation

    Engineering Animation

    1995

    International ANNIE Awards

    Finalist, received together with Walt Disney, for best animations in the film industry.

    Engineering Animation

    1996

    Industry Week magazine

    One of the 25 Technologies of the Year, for interactive 3D visualization and dynamics software used for product development.

    HelpMate Robotics

    1996

    Discover magazine

    One of 36 finalists for Technology of the Year, for the HelpMate robot used in hospitals.

    HelpMate Robotics

    1997

    Science Technology Foundation of Japan

    Japan Prize, to CEO Joseph Engelberger, for “systems engineering for an artifactual environment.”

    Illinois Superconductor

    1996

    Microwave & RF magazine

    One of the Top Products of 1996, for cellular phone site filters and superconductoring ceramics.

    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×

    Page 266

    TABLE 3 Continued

    Project Awardee

    Year

    Awarding Organization

    Award

    Illinois Superconductor

    1997

    American Ceramic Society

    Corporate Technical Achievement Award.

    Integra LifeSciences

    1999

    Thomas Alva Edison Award.

    Molecular Simulations

    1996

    Computerworld magazine

    Finalist for Smithsonian Award, the 1996 Innovator Medal.

    NCMS

    1994

    Institute for Interconnecting & Packaging Electronic Circuits

    Best Paper of Conference Awards.

    Outside Recognition

    There are a number of indicators that new knowledge has resulted from the R&D effort. 11 One that provides an early signal of the significance of the advance is recognition by outside parties. Table 3 lists awards that recognize the significance of technical accomplishments of this first group of 50 projects.

    Patents

    Patents, publications and presentations, as well as new and improved products and processes, not only indicate that new knowledge has been created, but also provide a relatively convenient means for others to acquire the knowledge. All but one of the 50 projects produced one or more of these outputs, indicating that they both created technical knowledge and provided a means of disseminating it.

    Citations

    Twenty-six of the 50 had filed a total of 115 patents at the time they were studied, 64 of which had been granted. And, other organizations were citing these patents, in some cases heavily. Patent citations are of particular interest because they signal the intensity of interest in an innovation and the intensity of its dissemination. Figure 1 illustrates a “patent tree” 12 for three of 12 patents granted to ETOM Technologies, Inc., an award recipient that went bankrupt soon after the end of its ATP project. It demonstrates that the knowledge created can survive and spread even if the award recipient fails, an important point of ATP's two pathways (direct and indirect) to impact.



    11 Assessing the degree of significance of scientific and technical knowledge gained from the projects is a daunting task. Even those projects that were not fully successful in achieving all of their research goals, or those that have not been followed by strong progress in commercialization, have achieved knowledge gains that may eventually prove to be of great value. Here the approach taken to suggest significance is to rely on outside recognition by other organizations that see value in the knowledge, to examine the use of the knowledge by others as signaled by patent citations, and to look downstream to its unfolding value in use.

    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×

    Page 267

    Image: jpg
    ~ enlarge ~
    FIGURE 1 Patent tree for the ETOM project: The knowledge is disseminating although the company went bankrupt




    12 Each published patent contains a list of previous patents and scholarly papers which establish the prior art as it relates to the invention. The citations provide a way to track the spread of technical knowledge through patents granted to ATP-funded projects. By following the trail of patent citations in subsequent patents back to the patents granted to the ATP-funded projects, it is possible to construct what looks much like an inverted genealogy tree. The patent trees were constructed under the direction of the project leader, Darin Boville of ATP's Economic Assessment Office.

    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×

    Page 268

    Publications

    Participants in more than half of the 50 projects published their research in technical and professional journals or presented papers in public forums, or both. Altogether, they published at least 180 papers and presented 245 or more papers, providing another way for others to learn about their research findings.

    Collaboration

    The extensive collaborative activities among these projects provided another avenue for the spread of new information. As was shown by Table 1, most (84 percent) of these projects entailed collaborations with others, including other companies, universities, national laboratories, non-profit consortia, and other organizations and individuals.

    New Products

    New and improved goods and services entering the marketplace also signal that the R&D has produced results, and, at the same time, often enable buyers to learn a great deal about the technology. For more than 60 percent of the projects, commercial products or processes based on the ATP-funded technology had reached the market, providing others with the ability to collect information about the new technologies through use, examination, and reverse engineering.

    Public Meetings

    The ATP organizes and sponsors numerous public workshops, where companies present non-confidential aspects of their ATP-funded research and engage in open discussions. These workshops facilitate information flows in several directions—among ATP award recipients and from them to other companies, ATP project managers, other government program managers, the press, potential investors, and universities. The ATP also makes project information available on its internet web site ( http://www.atp.nist ), and adds new non-proprietary project descriptions as new awards are made. Evaluation reports, such as this one, are an additional source of information to the public about the projects.

    COMMERCIALIZATION: PUTTING KNOWLEDGE TO USE

    If the new knowledge is to yield economic benefits to the nation, the award recipients who develop it, their collaborators, or other companies who acquire it, must put it to use. A second focus of this study is on commercialization progress of the award recipients, that is, progress along ATP's direct path of project impact. This is the path by which ATP is best able to accelerate the use of the

    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×

    Page 269

    technology by U.S. companies, the indirect path of knowledge dissemination being less subject to influence.

    Companies in 39 of the 50 projects either had one or more products or processes in the market or expected to shortly. Table 4 shows that 80 total products and processes were commercialized already or expected soon.

    Although commercializing a technology is an important step, it does not mean that the project is necessarily headed toward full success from the perspective of either the company or the ATP. In some cases products have been sold in relatively small number for testing and evaluation, and do not yet have a significant market. In several cases, struggling companies made small sales and subsequently failed due to cash-flow problems. Moreover, the early products and processes often embody partial aspects of the technology, as companies—particularly small ones—hurry to establish a revenue stream that they need for survival, while they continue to work toward larger objectives.

    RealCommercial Products

    Whether or not widespread diffusion of a technology ultimately results from commercial activities of award recipients remains to be seen in most cases, but it is highly significant that products and processes are actually on the market. This is an extremely important step for the eventual generation of broad-based economic benefits from the new technical capabilities. The bottom line is that despite the difficulty of moving from the research stage to commercialization, companies in 78 percent of the 50 projects either sold product, or used or licensed to others process improvements stemming from their research, or they were about to do so at the time they were contacted by study analysts.

    Rapid Growth

    An indicator that a small research-oriented company is on the path toward commercialization is company growth. A recent look at Fortune's Fastest Grow

    TABLE 4 Progress of Participating Companies in Commercializing the New Technologies

    Product/Process on theMarket

    First Product/Process ExpectedSoon

    On the Market with Additional Product/Process Expected Soon

    On the Market or Expect Soon—Totals

    Number of Projects

    32

    7

    9

    39

    Number of Products/Processes

    61

    9

    10

    80



    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×

    Page 270

    ing 100 Companies List found two of the 31 then-small companies on the list. Capitalized value of some of the companies has increased by hundreds of millions of dollars. Nearly a fifth grew their employment by more than 500 percent from the beginning of the project to several years after the project had completed, and 61 percent grew employment by more than 100 percent. 13 Several of the companies that were small when they received the ATP award, have grown out of that size category. Nineteen of the 31 small companies at least doubled in size; four companies grew more than 1,000 percent.

    Not all the small companies grew—a little more than a fifth experienced no change or decreases in staff—but, as a group, the small companies funded by ATP grew rapidly as they parlayed their new technical capabilities into business opportunities.

    OVERALL PROJECT PERFORMANCE

    Figure 2 shows how the projects performed overall, where continuing progress is necessary for a high score. 14 At a glance, we can see from the ratings that the majority of the projects are still alive, in the sense that progress has been made and further progress appears likely. Sixteen percent of the projects were top performers in terms of overall project effectiveness. Fifty-eight percent were in the middle group. Twenty-six percent received low scores.

    For these projects, progress signaled by the performance metrics is translating into benefits. Keeping in mind that the top scores represent a minority of the awardees, let us consider the estimated benefits of projects among the group of top scorers.

    Medical Cost Savings

    Three of the top performers developed medical technologies that were evaluated by economists at the Research Triangle Institute (RTI), a consulting firm in North Carolina. 15 RTI economists provided early estimates of the value of a new biopolymer to repair fractures, developed by Integra LifeSciences, a system for replicating stem cells, developed by Aastrom Biosciences, Inc., and a new prosthesis material—animal-derived extra-cellular matrix, or ADMAT, developed by Tissue Engineering, Inc.



    13 Employment changes in joint ventures, larger companies, and non-profit organizations are less closely tied to the success of individual research projects, and, their employment was not tracked.

    14 The composite overall scores were computed using all the raw inputs of knowledge creation and dissemination and commercialization progress. The latter were weighted more heavily such that only projects with continued progress by award recipients receive top scores.

    15 For a more detailed treatment of the RTI study see Taylor H. Bingham, “ Estimating Economic Benefits from ATP Funding of New Medical Technologies,” in this volume.
    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×

    Page 271

    Image: jpg
    ~ enlarge ~
    FIGURE 2 Distribution of projects by overall performance score

    Biopolymer Repair

    The RTI study estimated the medical cost savings from Integra's project in terms of avoiding second surgeries to remove implants, such as pins and screws, when trouble arises. It estimated benefits at $98 million, all attributed to the ATP. 16 Integra has shown continued robust progress since the RTI study was conducted. At the time the analysis of the company for this mini-study was completed in 1999, company employment had increased four-fold, and a recent check showed another big increase. Through its commercial partners, Integra's technology is becoming embodied in surgical screws, tacks, and other fixation devices for attaching soft tissue to bone in the knee and shoulder. Pending FDA approval, patients are positioned to benefit in terms of complications and further surgeries avoided, as well as from lower surgical costs.

    Stem Cell Expansion

    Because of the difficulty of estimating the value of patient pain reduction and improved health outcomes, the RTI study also based the benefits estimates for Aastrom's stem cell expansion technology only on the reduction in procedure cost, although these former effects are the main expected feature of this technology. RTI economists estimated that the replication system, once implemented, would save about $87 million (in 1997 dollars) in the costs of providing bone-



    16 The estimate, which was net of costs, did not include benefits of patient health effects and pain-avoided.
    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×

    Page 272

    marrow transplants for cancer treatment without the acceleration provided by ATP support and $134 million with the acceleration. The difference, $47 million, is the estimated additional value, in terms of cost savings only, attributed to the ATP, based on this application area. Aastrom, the award recipient, has since continued unabated pursuit of the commercialization of its AastromReplicall™ System. In recent clinical trials, the system was used successfully to enable cancer patients for which there were otherwise no donors to receive stem cell transplants. According to the director of medical oncology at Hackensack University Medical Center, “These results suggest that we may have found a new treatment approach that will enable more patients to receive treatment for this very serious and often fatal disease.” 17 According to the American Cancer Society, 30,000 new cases of leukemia are expected in 2000 and approximately 20,000 people will die from the disease this year. More effective treatments able to reduce this toll would be of great value to society.

    Repairing Knees

    Focusing on the first expected application of Tissue Engineering's ADMAT technology, namely, the repair of damaged knee ligaments (specifically, anterior cruciate ligaments, or ACLs), RTI economists also estimated benefits. But in this case, unlike the previous two, RTI was able to estimate benefits expected to result from improvements in the quality of life for patients receiving the treatment, by using a “quality-adjusted-life-years” index value. 18 RTI estimated about $15 billion in expected net benefits from the new technology attributable to ATP funding.

    Printed Wiring Board

    Another of the top performing projects is a joint venture led by the National Center for Manufacturing Sciences (NCMS) to develop a suite of advanced technologies for producing printed wiring boards (PWBs), the backbones of electronics products. Two studies conducted by Professor Albert Link of the University of North Carolina-Greensboro, focused on assessing the impact of the project's extensive use of collaborative effort. 19 The studies estimated that the project's collaborative effort produced at least a 53 percent reduction in overall research costs, resulting in an R&D savings of at least $35.5 million, in the process of producing new capabilities that the industry needed for international competitive

    17 PRNewswire, 19 April 2000, report on results from Hackensack/Aastrom Studies.

    18 For a description of the use of Quality-Adjusted Life-Years (QALY) in evaluating patient benefits in evaluation studies, see A. J. Wang, “Key Concepts in Evaluating Outcomes of ATP Funding of Medical Technologies,” The Journal of Technology Transfer, 23(2):61-66.

    19 For a more detailed treatment see A. N. Link “Enhanced R&D Efficiency in an ATP-funded Joint Venture,” in this volume.

    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×

    Page 273

    ness. The project has yielded productivity improvements for member companies and improved competitive positions of U.S. suppliers in the world market for PWBs. 20 Award-winning papers, new products, and other knowledge dissemination activities by the joint venture has helped to spread the new capabilities across the entire industry. At a recent technology exposition by ATP-funded companies, an advanced circuit board was displayed that incorporated many of the innovations developed by the ATP-funded project. A small company, one that did not participate directly in the project, produced it, suggesting the flow of knowledge from the project to others. 21

    Improved Software to Process High-volume Data

    Another of the top performers is a project led by Torrent Systems, Inc., which has developed a component software system that insulates programmers from the complexities of parallel programming while allowing them to use it productively in scalable applications. Torrent delivered this new capability in its software product, Orchestrate™. An early company user of the new software reportedly was able to increase its revenue by $100 million per year. 22 Torrent's technology is making it possible for eBusinesses and other companies to process and analyze great volumes of data. In addition to receiving a number of other awards recognizing its software technology, Torrent was listed in ComputerWorld's “100 Hot Emerging Companies” in 1998.

    High-Temperature Superconducting Wire

    American Superconductor Corporation, who led another of the top performing projects, is producing high-temperature superconducting wire for use by electric utilities and as a component in motors, transformers, and specialty magnets to reduce their energy consumption. With an estimated sales volume of $15 million in 2000, and a rapid sales growth rate, this small company is commercializing its technology. The technology's ability to reduce energy costs has taken on increased significance in the face of rising energy prices.

    Improved Medical Procedures

    Like the preceding examples, Engineering Animation, Inc., leader of another of the top performing projects, has aggressively and successfully pursued appli




    20 The president of NCMS credited the ATP project with saving the PWB industry in the U.S. with its approximately 200,000 jobs.

    21 ATP Technology Showcase, “Ten Years of Innovation and Impact,” discussions with an NCMS spokesperson, hosted by the Advanced Technology Program, Washington, D.C.: Russell Senate Building, Caucus Room, 5 April 2000.



    22 Information from Hoover's on-line company search and Torrent's web site, current 31 August 2000.


    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×

    Page 274

    cations of its award-winning imaging software capabilities developed in the ATP-funded project. The company used its ATP-funded technology to improve the training of doctors, as well as to guide medical procedures. Patients in a particular surgical procedure that employed the company's visualization software reportedly had better outcomes as a result. Founded by two professors and two graduate students in 1990, the company had 20 employees at the time it received the ATP award. The company now employs approximately 1,000, had sales of $71 million in 1999, and experienced a sales growth rate over the past year of 34 percent. According to company officials, the ATP award allowed it to significantly extend its capabilities in computer visualization and computations dynamics and to form important collaborative relationships that it has since been able to leverage in many different directions. Recently, it has extended and deployed its awardwinning visualization capabilities to develop a virtual factory technology, implemented recently at Ford, which enables faster design and analysis of factory models. Its many customers and clients have in turn benefited.

    Better Auto Bodies

    To these examples, we can add a number of other strong projects from among the 50 that produced technologies that are delivering important benefits. One such project, led by the Auto Body Consortium, has generated documented production cost savings and improved automobile quality, as well as the potential for extending these same kinds of benefits to the manufacturing assembly of other products. A study by the CONSAD Research Corporation attributed economy-wide benefits of about $3 billion in the year 2000 to the project. 23

    …and Better Enzymes

    Another example is provided by a project led by Amersham Pharmacia Biotech, which is credited with accelerating development of an enzyme important to the speed of the human genome project—where timing is of enormous significance. To these examples, we can also add a number of other promising technologies—technologies that may improve productivity, facilitate better weather forecasts, improve communications, enable new drug discovery, reduce energy costs, and lower loss of limb and life globally by improving detection of old land mines and toxins.

    23 CONSAD Research Corporation, Advanced Technology Program Case Study: The Development of Advanced Technologies and Systems for Controlling Dimensional Variation in Automobile Body Manufacturing, NIST GCR 97-709, March 1997. A new study is currently underway, led by MIT researchers, to extend and update the analysis of this project. Contact ATP's Economic Assessment Office for further information.

    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×

    Page 275

    WHAT CONSTITUTES SUCCESS AND FAILURE FOR ATP?

    Because individual-project failure must be anticipated and tolerated in a program that focuses on overcoming challenging technical barriers to innovation, it is essential to take a project-portfolio approach to assessing the ATP against the legislated mission of the program.

    Three general tests, and several additional specific tests—all derived from ATP's mission—if applied after sufficient passage of time, should reveal the extent to which ATP has successfully met its mission: 24

    • Test 1: Has the portfolio of ATP-funded projects overall produced large net benefits (i.e., benefits minus costs) for the nation?

    • Test 2: Have a substantial share of the net national benefits accrued to citizens and organizations beyond the ATP-award recipients?

    • Test 3: Did ATP make a substantial positive difference in the size and timing of the benefits?

    Additional specific tests of success include the following: Did the projects produce new scientific and technical knowledge? Did the ATP increase collaboration? Were small businesses able to participate? Were manufacturing capabilities improved? Did U.S. businesses become better able to compete in global markets?

    Partial Answers

    While the ultimate answers to these success “test questions” depend on the long-run impacts of the portfolio of ATP projects, the performance-to-date of the sub-portfolio of 50 projects provides partial answers of considerable interest.

    The performance ratings show that the majority of the projects are still underway, and further progress continues to be made. More importantly, they reveal a core group of highly active and productive performers, albeit a minority, who are successfully accomplishing the big goals of their projects.

    The ATP awarded a total of $104.0 million to the 50 completed projects. What is the public investment producing in the way of benefits? The ATP's impact appears to be substantial from the perspective of economic return on the government expenditure (i.e., the cost of the award), the catalytic effect of the award (i.e., its ability to bring technology forward and leverage previous funding), and through the substantial downstream positive spillovers or benefits to others.

    24 The tests of success are taken from a presentation by Rosalie T. Ruegg before the National Grants Management Association's Annual Conference, Federal Bar Association Panel, 4 April 2000.

    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×

    Page 276

    Economic Returns

    Estimated quantifiable economic benefits attributed to the ATP from just a few of the top performing 50 projects not only greatly exceed ATP's funding for all of the 50 projects, they also far exceed the total of ATP costs for all of the 522 projects funded to date.

    Catalytic Effects

    Detailed case studies and extensive interviews with the project leaders attributed to ATP a key role in bringing the projects to fruition or in accelerating their development.

    Social Benefits

    In addition to benefits exceeding costs, there is strong evidence that benefits are extending well beyond those captured by the award recipients. There is substantial evidence that the knowledge generated by the projects is being disseminated through publications, presentations, patents, products, and other means. The patent trees developed for these projects reveal rich citations of the patents by others. The products and processes generated by the projects are also yielding benefits extending beyond project participants. For example, patients are receiving spillover benefits from new, better medical treatments available at lower costs; consumers are receiving spillover benefits when they buy superior products for which they pay less than the full value; and companies outside the projects are receiving spillovers when they increase their productivity or achieve greater value added by using ATP-funded technologies.

    These answers are, as we underscore, partial in nature. More time is necessary for a fuller assessment. Over time, a fuller assessment should reveal wider diffusion of some technologies, the demise of others, perhaps many others, and major success from a few. What this research does show is that the program is achieving its goals for a significant number of awards. It also highlights the research and measurement efforts of a program able and willing to document its failures as well as its successes. The failures in part offer reassuring evidence that the program continues to pursue its mandate of investing in high-risk, but potentially high-payoff, technologies with gains for participants and for society.

    REFERENCES

    Bingham, Taylor H. 2001 . “Estimating Economic Benefits from ATP Funding of New Medical Technologies.” in this volume.

    CONSAD Research Corporation. 1997 . Advanced Technology Program Case Study: The Development of Advanced Technologies and Systems for Controlling Dimensional Variation in Automobile Body Manufacturing . NIST GCR 97-709.

    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×

    Page 277

    Link, Albert N. 2001 . “Enhanced R&D Efficiency in an ATP-funded Joint Venture.” in this volume.

    Long, William F. 1999 . Performance of Completed Projects: Status Report Number 1 . NIST SP 950-1. March.

    PRNewswire. 2000 . Report on results from Hackensack/Aastrom Studies . April 19.

    Ruegg, Rosalie, et al. Forthcoming. Performance of 50 Completed Projects: Status Report Number 2 , NIST SP 950-2.

    Ruegg, Rosalie. 2000 . Presentation before the National Grants Management Association's Annual Conference, Federal Bar Association Panel . April 4.

    U.S. General Accounting Office. 1996 . Effectively Implementing the Government Performance and Results Act . GGD-96-118. Washington, DC : General Accounting Office .

    Wang, Andrew. 1998 . “Key Concepts in Evaluating Outcomes of ATP Funding of Medical Technologies.” The Journal of Technology Transfer . 23(2): 61-66 .

    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×

    Page 278

    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×
    Page 259
    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×
    Page 260
    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×
    Page 261
    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×
    Page 262
    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×
    Page 263
    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×
    Page 264
    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×
    Page 265
    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×
    Page 266
    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×
    Page 267
    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×
    Page 268
    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×
    Page 269
    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×
    Page 270
    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×
    Page 271
    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×
    Page 272
    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×
    Page 273
    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×
    Page 274
    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×
    Page 275
    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×
    Page 276
    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×
    Page 277
    Suggested Citation:"Taking a Step Back: An Early Results Overview of Fifty ATP Awards." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
    ×
    Page 278
    Next: Annex A. Authorizing Legislation for the Advanced Technology Program »
    The Advanced Technology Program: Assessing Outcomes Get This Book
    ×
    Buy Paperback | $80.00 Buy Ebook | $64.99
    MyNAP members save 10% online.
    Login or Register to save!
    Download Free PDF

    This report examines the operations of the APT, reviews its extensive assessment program, and provides NRC Committee findings concerning the ATP’s operations and recommendations for potential improvements to the program. The report includes a summary of a major conference held in April 2000 as well as seven papers, including surveys of the industry participants or users of the ATP program, a summary of the results of fifty awards, detailed assessments of major joint ventures, and a description of the current selection process. It is the most comprehensive study to date of the program’s origins, operations, achievements, and assessment. Its conclusion: the program works.

    1. ×

      Welcome to OpenBook!

      You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

      Do you want to take a quick tour of the OpenBook's features?

      No Thanks Take a Tour »
    2. ×

      Show this book's table of contents, where you can jump to any chapter by name.

      « Back Next »
    3. ×

      ...or use these buttons to go back to the previous chapter or skip to the next one.

      « Back Next »
    4. ×

      Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

      « Back Next »
    5. ×

      Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

      « Back Next »
    6. ×

      To search the entire text of this book, type in your search term here and press Enter.

      « Back Next »
    7. ×

      Share a link to this book page on your preferred social network or via email.

      « Back Next »
    8. ×

      View our suggested citation for this chapter.

      « Back Next »
    9. ×

      Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

      « Back Next »
    Stay Connected!