National Academies Press: OpenBook

The Advanced Technology Program: Assessing Outcomes (2001)

Chapter: Summary of Symposium Proceedings

« Previous: Overview of the Papers
Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×

Page 63

C. SUMMARY OF SYMPOSIUM PROCEEDINGS

The text below provides a summary of the major statements by participants in the symposium.

Introduction to the Symposium

Clark McFadden, a partner in the Washington law firm Dewey Ballantine, opened the symposium with a review of the policy issues facing the ATP. Mr. McFadden, who has broad experience in the creation of government-industry partnerships, is a member of the Steering Committee assembled by the STEP Board to carry out its program-based review of government-industry partnerships. He noted that this fact-finding meeting is one in a series of reviews organized under the auspices of the NRC program on Government-Industry Partnerships for the Development of New Technologies.

The ATP, he said, is unique among government research and technology programs in several respects:

  • The program's goals go beyond basic research (or specific mission goals) to supporting industrial research in “enabling” technologies and encouraging commercialization of those technologies. It is therefore on the leading edge of government-industry relationships.

  • The program has been intensively studied and assessed. The core competence of the National Institute of Standards and Technology, ATP's parent, is measurement, and the agency has assiduously assessed the outcomes of the program. 124 The Steering Committee, for its part, has been able to draw on the extensive assessment effort that NIST has commissioned in cooperation with economists of the independent National Bureau of Economic Research (NBER) and other researchers. The Committee's goal is to bring varied points of view to bear on the program's effectiveness, efficiency, and rationale. 125 These reviews have helped the Steering Com-





124 Several recent examples are included in this volume, such as D. Austin and M. Macauley, Estimating Future Consumer Benefits from ATP-funded Innovation: The Case of Digital Data Storage, NIST GCR 00-790, April 2000 (see
Chapter V of this volume). The ATP assessments also include a report by William F. Long providing a comprehensive, qualitative review of the outcomes of ATP investments, documenting research accomplishments, subsequent work by companies to commercialize the results, and near-term outlooks for the successful companies. This rich summary is exceptional in that it also documents ATP projects that were terminated in the same period, with brief explanations. William F. Long, Advanced Technology Program: Performance of Completed Projects—Status Report Number 1, op. cit. Rosalie Ruegg provides an updated overview of progress on fifty awards in
Chapter V of this volume.

125 In addition to NIST's own research, the U.S. General Accounting Office and congressional committees have shone their own spotlights on the program. The findings of several of these reports are summarized above. For the complete reports, see U.S. General Accounting Office, Advanced


Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×

Page 64

mittee develop better ways to balance the risks, returns, and opportunity costs of other government-industry collaborations.

The Steering Committee, McFadden added, has already produced one publicly available report on the ATP, describing its goals, assessment program, and providing comments from critics, supporters, and participants in the program. 126 The focus of today's meeting, he said, is to assess the program's results and its effort at evaluation.

The ATP Objective: Addressing the Financing Gap for Enabling Technologies

Committee member Charlie Trimble, of Trimble Navigation, introduced the first panel by reminding participants that the ATP was originally established to address the “funding gap” for emerging technologies (that is, the tendency of the private sector to underinvest in early-phase research and development). This session would attempt to understand the behavior of private investors and their perspective on the ATP, through the prism of three panelists, from widely different worlds:

  • the proprietor of a start-up company that has received ATP funds;

  • a successful venture capitalist; and

  • a representative of a global technology firm that participates in the program.

A Start-up's Perspective

Elizabeth Downing—President and CEO of a Palo Alto high-technology start-up company—reviewed the experience of her firm, which had received a single-company ATP grant about one and one-half years earlier to develop a new concept in imaging known as Crossed-Beam Displays. Addressing a fundamental question about the ATP, she asked, “Why should government fund the development of enabling technologies? Because enabling technologies have the potential to bring enormous benefits to society as a whole. Yet private investors will not





Technology: Proposal Review Process and Treatment of Foreign-Owned Businesses, RCED-94-81, Washington, D.C.: U.S. Government Printing Office, 1994; U.S. General Accounting Office, Performance Measurement: Efforts to Evaluate the Advanced Technology Program, op. cit.; U.S. General Accounting Office, Measuring Performance: The Advanced Technology Program and Private-Sector Funding, op. cit.; U.S. General Accounting Office, Federal Research: Information on the Advanced Technology Program's 1997 Award Selection, op. cit.; U.S. General Accounting Office, Advanced Technology Program: Inherent Factors in Selection Process Could Limit Identification of Similar Research, op. cit.

126 National Research Council, The Advanced Technology Program: Challenges and Opportunities, op. cit.

Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×

Page 65

adequately support the development of these technologies, because profits are too uncertain or too distant.”

Enabling technologies, she said, are technologies that show the potential to make radical improvements in some aspect of society or even create new industries. However, they tend to be disruptive—by displacing entrenched industries. Fundamentally, they are too immature and risky to interest private investors, which is understandable, because they may require years or decades to become commercially viable and generate profits. 127

Another unusual feature of her company's experience is that it has received several different types of federal awards. These include SBIR grants, DARPA contracts, and Cooperative Research and Development Agreements with universities. Of all the federal funding programs that her young company has participated in, she said, the ATP was by far the most useful because of its large funding commitment, long time duration, and administrative simplicity. It allows the technical and market risks of the technology to be reduced, while keeping control of the technology in the hands of the company itself.



Box H.Why Should Government Fund Promising Technologies?

“Why should government fund the development of enabling technologies? Because enabling technologies have the potential to bring enormous benefits to society as a whole. Yet private investors will not adequately support the development of these technologies, because profits are too uncertain or too distant.”

— Elizabeth Downing, 3D Technology Laboratories



A Venture Capitalist's Perspective

The next speaker, a past president of the National Venture Capital Association, brought the perspective of the venture capital community to the goals of the ATP. David Morgenthaler, founder of Morgenthaler Venture Capital, observed that venture capital investments have grown dramatically in recent years, from $4.9 billion in 1993 to almost $50 billion in 1999. But, he continued, venture capitalists must be accountable to their investors, who demand a positive and fairly predictable internal rate of return, within a limited period (usually five years or less). In response to the classic question “At what stage in the maturity of a



127 For a discussion of this key question see L. M. Branscomb, K. P. Morse, and M. J. Roberts, Managing Technical Risk: Understanding Private Sector Decision Making on Early Stage Technology-based Projects, NIST GCR 00787. April 2000. See also L.M. Branscomb and Philip E. Auerswald, Taking Technical Risks, op. cit.

Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×

Page 66

technology should a venture capitalist invest?” the answer is clearly “When technical uncertainties are reduced.” The enabling technologies in which the ATP typically invests represent too high a risk for private investors alone and are normally at too early a stage in their development.

“The venture capital business can be likened to a horse race,” he said, “in which the technology is the horse, the management of the company is the jockey, the market is the race, and the venture capitalist is the owner and trainer. What we want is a great horse and a great jockey and to enter them in the Kentucky Derby (a huge market, in our homely analogy).” The ATP creates better horse races by financing both more discoveries and more enabling technologies, which lead to new and better products and services.

A Large Company Perspective

Kathleen Kingscott, Director of Public Policy for Science and Technology at IBM, noted that the process of research and development has changed in the past 20 years, with much greater emphasis on cooperation. Even a leading technology company like IBM, with an annual R&D investment of $6 billion, cannot go it alone anymore. Collaboration is increasingly vital, and IBM finds the ATP an excellent source of new technology and collaborators. Partnerships are sought after because they are a key means of reducing costs and risks and provide access to new, promising technologies. They also make technology transfer more efficient by improving communication among researchers and managers.

In the discussion that followed these presentations, David Morgenthaler pointed out that venture capitalists tend to specialize in particular areas of technology because they must compete to recruit the best young companies. A record of achievement in the appropriate field can make the difference.

Noting that the ATP's practice of making multiple awards to single companies has been criticized, Maryellen Kelley of the ATP's evaluation office asked Kathleen Kingscott about IBM's experience. Ms. Kingscott replied that IBM has participated in two ATP grants as a single company and seven as a member of joint ventures, with a shifting but overlapping set of partners.



Box I. A Venture Capitalist's Perspective on the ATP

“[The ATP] is an excellent program for developing enabling, or platform, technologies, which can have broad applications but are long-term, risky investments. Venture capitalists are not going to fund these opportunities, because they will feel that they are at too early a stage of maturity. Government can and should fund these technologies. In fact, it should do more than it is doing.”

— David Morgenthaler, Morgenthaler Venture Capital

Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×

Page 67

The ATP's Assessment Program

David Goldston, Legislative Director for Representative Sherwood Boehlert, introduced this panel, which considered the technical details of the ATP's assessment program. Rosalie Ruegg, Director of the Economic Assessment Office of the ATP, provided an overview of the ATP assessment program, followed by Irwin Feller who commented on the utility of economic assessment in federal programs such as the ATP.

From 1990 through 1999, Ms. Ruegg said, the ATP has co-funded 468 projects, with 1,067 participants and another 1,027 subcontractors. More than half of the projects are led by small businesses. More than 145 universities participate, as do more than 20 National Laboratories. 128 Funding of these projects by the ATP and industry has totaled about $3 billion, with each contributing about half. Of all federal technology programs, the ATP is probably the most given to rigorous economic assessment, principally through contracts with the National Bureau of Economic Research. An overview of the ATP's evaluation program can be found in the report summarizing the STEP Board's earlier meeting on the ATP. 129

Measuring Impacts: Direct and Indirect Paths

The economic impacts of ATP projects can be measured in several ways, such as productivity gains, new businesses created, employment benefits, increases in gross domestic product (GDP) and improvements in the quality of life. The impacts of publicly funded R&D include both private returns to the company involved in the project and “spillovers” (impacts on others, including the company's customers, its competitors, and others who derive benefits or costs from the project's results).

The ATP traces these impacts along direct and indirect paths. The direct path follows the awardees and includes private returns to the particular companies directly involved in the ATP-funded projects and spillover benefits to their customers. The indirect path, no less important, involves the take-up of the knowledge generated by a project by others outside the project who have not directly contributed to the investment cost. Even if a project fails to find a commercial market—even if, in David Morgenthaler's analogy, the jockey falls off the horse—these indirect impacts continue to circulate through society as the knowledge is picked up and exploited by others. All of the indirect impacts can be considered spillovers from the original R&D. The impacts along the direct and indirect paths combined represent what economists call the social return of the project.





128 Through the end of 2000, the ATP funded 522 projects with 1,162 participants and an approximately equal number of subcontractors. Through that time, 176 universities had participated.

129 National Research Council, The Advanced Technology Program: Challenges and Opportunities, op. cit. pp. 71-80.

Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×

Page 68

Assessment Tools

To track these impacts, the ATP uses a variety of statistical tools, including

  • statistical profiling of applicants, projects, participants, and technologies;

  • progress tracking of all projects and participants (through a business reporting system and other surveys);

  • status reports for all completed projects;

  • detailed microeconomic case studies of selected projects and programs;

  • econometric and statistical studies of innovation, productivity, and portfolio impacts;

  • limited use of macroeconomic analysis for selected projects;

  • special issue studies; and

  • development and testing of new assessment models and tools.

Ms. Ruegg also gave a short overview of the results of the evaluation of the first 50 completed projects, pointing out that a small number were terminated for a variety of reasons, the largest number were making progress but were as yet inconclusive, and a significant proportion of the projects were extremely promising.

Box J. Advancing the Art of Program Assessment

“ATP has gone beyond the efforts of other programs that have sought to measure direct benefits, by trying to measure indirect or “spillover” benefits. Measuring these impacts is a difficult, if not heroic, task. ATP's assessment techniques are at the state-of-the-art and, in many ways, have advanced the state of the art.”

— Irwin Feller, Pennsylvania State University

Integrating “State-of-the-Art” Assessments

Economists' “irrational passion for dispassionate rationality,” said Irwin Feller, of Pennsylvania State University, leads them to scrutinize dispassionately the impact of dispassionate rationality on public policy making—which has often been characterized by passionate irrationality.

In reviewing the ATP evaluation process, Dr. Feller commented on three topics: (a) technical aspects of the evaluation process itself, (b) use of the findings of that evaluation within the ATP, and (c) use of the findings of evaluation outside of the ATP.

Dr. Feller noted that the ATP has much to be proud of in its approach to assessment, compared, for example, with NIST's Manufacturing Extension Part

Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×

Page 69

nership Program (MEP). The ATP has gone beyond the efforts of other programs that have sought to measure direct benefits, by trying to measure indirect or “spillover” benefits. Dr. Feller remarked that the ATP's assessment techniques are clearly at the state-of-the art and in many ways have advanced the state of the art. Despite this accomplishment, a key question for the program is how effectively does the ATP use the results of the evaluation in its internal operations, notably in its project selection criteria?

A related issue is the impact of these excellent evaluations on decision makers outside the ATP, particularly in Congress. The ATP seems to be chronically beset by opposition in Congress to its very existence, and its budget tends to be erratic. Dr. Feller noted that Tolstoy once said, “Doing good will not make you happy, but doing bad will surely make you unhappy.” The evaluator's corollary is “A good evaluation showing bad results will surely kill a program, but an evaluation that shows good results may not save a program.” In closing, he observed that the real challenge is how to convert the ATP over time into a credible, institutionalized part of the federal science and technology apparatus, and thereby defuse the ideological objections of its opponents. If the program does well over time, this can happen. In any case, the ATP assessment program itself is a model for other partnership activities. As a dispassionate rationalist, he said, he would like to believe that it will be persuasive over time.

Mr. Goldston introduced the two discussants for the panel: Dr. Nicholas Vonortas of George Washington University and James Turner of the minority staff of the House Science Committee. Dr. Vonortas retraced the history of technology policy over the past two decades, in which he said that federal attempts to solve the “competitiveness” problem began as rather simplistic attempts to imitate the Japanese government and industry. In the end, though, these initiatives—and in particular the ATP with its serious approach to evaluation—have had the salutary effect of merging the two previously distinct literatures of business and economics. He also observed that the major European countries are more comfortable with programs of this kind, perhaps because in the past they relied less on the defense model. 130

Solid Evaluations—Unstable Budgets

Jim Turner returned to the observation that the ATP is far ahead of other federal programs in its use of solid program evaluation. At the same time, as Dr. Feller had pointed out earlier, it remains in an unstable budgetary position in Congress. The contrast with the MEP is particularly striking. Both programs were

130 Programs to support high-technology industry are widespread in Europe. Although often criticized as cumbersome or ineffective, over time regional and national programs to support industries such as semiconductors have had substantial positive impacts. See Thomas R. Howell, “An Overview of Government Policy Measures in Microelectronics,” op. cit.

Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×

Page 70

founded under the same statute (the 1988 Omnibus Trade and Competitiveness Act). The MEP was designed to have a strong base in each of the fifty states and therefore quickly became politically invulnerable. With its rigorous, apolitical approach and merit-based review, the ATP has never enjoyed that deep and broad support in the Congress.

Questions from the audience provoked a lively discussion of the political dilemma of the ATP. Dr. Lewis Branscomb endorsed Dr. Feller's comments on the uncertain link between positive economic evaluation and political support. Dr. Branscomb suggested that political evaluation as well as economic evaluation is needed. 131 He also suggested that few federal agencies practice intellectually sophisticated political evaluations.

David Goldston agreed that the ATP is well managed. Still, he said, many in Congress have fundamental concerns about the role of the federal government in the ATP, which are at the crux of one of the difficult—perhaps insoluble—questions about the ATP. Would the projects have been done anyway without the ATP? Are there more productive uses of federal money?

Jim Turner, Charlie Trimble, and David Goldston discussed the failure rate of the ATP projects. They agreed that too low a failure rate would suggest that the program was not tackling risky enough projects. At the same time, a high failure rate may be politically risky. It was pointed out that the ATP does terminate poorly performing projects—a good, if unusual, feature for a federal program.

Rosalie Ruegg assured Dr. Feller that the ATP management had taken steps to use the results of its evaluation to improve its selection process.

James Turner closed the discussion on a political note, observing that—ironically, perhaps—the drafters of the ATP's authorizing legislation had intended the program to be a political bolster for NIST, its parent agency. They had hoped that it would quickly develop a presence in all fifty states, something that has not yet occurred, as the chart below shows.

Stimulating R&D Investment

David Finifter, of the College of William & Mary, introduced this panel with the observation that, as an economist, he believed that the day's proceedings were making it clear that the ATP has greatly advanced the art of program evaluation.

Views from ATP Applicants

The first researcher was Dr. Maryann Feldman of Johns Hopkins University, who reviewed her study of the differences between winners and non-winners in

131 For an informative discussion of U.S. technology policy, see L. M. Branscomb and J. Keller, editors, Investing in Innovation, op. cit. For a discussion of the ATP in comparison with the SBIR and other elements of the early-phase financing of new technologies, see L. M. Branscomb and P. E. Auerswald, Taking Technical Risks, op. cit.

Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×

Page 71

ADVANCED TECHNOLOGY PROGRAM

Ongoing/Completed Projects

Project-Level Award Amounts ($M), Summed by State

(Forty-One Competitions: 1990-2000)

STATE

NUMBER OF PROJECTS *

ATP AWARDS ($M)

INDUSTRY-SHARE ($M)

TOTAL($M)

Alabama

1

$3.3

$3.5

$6.8

Arizona

5

$16.6

$14

$30.6

California

120

$360.7

$353.6

$714.3

Colorado

8

$15

$8.5

$23.5

Connecticut

19

$55.3

$55.5

$110.8

Delaware

5

$9.4

$7.6

$17

Florida

7

$28.7

$29.8

$58.5

Georgia

6

$12.3

$7.2

$19.5

Illinois

21

$71.3

$75.7

$147

Indiana

2

$3.6

$3.2

$6.8

Iowa

2

$2.6

$1.4

$4

Louisiana

2

$3.8

$3.1

$6.9

Maryland

16

$50

$45

$95

Massachusetts

48

$96.2

$78.1

$174.3

Michigan

41

$182.4

$192.2

$374.6

Minnesota

17

$60.9

$70.3

$131.2

Missouri

1

$2

$1.4

$3.4

Nebraska

1

$2

$0.9

$2.9

New Hampshire

2

$4

$1

$5

New Jersey

26

$88.1

$95.5

$183.6

New Mexico

1

$2

$1.8

$3.8

New York

29

$72.1

$73.7

$145.8

North Carolina

7

$34.4

$33.1

$67.5

Ohio

17

$70.6

$71.6

$142.2

Oklahoma

2

$3.5

$3

$6.5

Oregon

8

$18.9

$17.7

$36.6

Pennsylvania

18

$57.1

$61.8

$118.9

Rhode Island

3

$4.4

$2.6

$7

South Carolina

3

$41.4

$48

$89.4

Texas

18

$59.7

$53.1

$112.8

Utah

8

$15.2

$12.9

$28.1

Virginia

10

$31.1

$23.3

$54.4

Washington

2

$3.9

$1.4

$5.3

Wisconsin

5

$9

$6.1

$15.1

State Count

Project Count

Total ATP ($M)

Total Industry ($M)

Grand Total ($M)

34

481

$1,491.5

$1,457.6

$2,949.1



*Project counts are based on location of “Lead” organization (single applicants/joint venture lead organizations).


Note: Award and cost-share amounts listed for joint ventures represent amounts awarded to the joint venture as a whole. Not all participants in these joint ventures are necessarily located in one state—many other
joint venture participants in a particular state may participate in joint ventures led by organizations in other states.




Source: NIST. Terminated projects are excluded.

Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×

Page 72

the 1998 ATP competitions. 132 Dr. Feldman began her presentation by calling for a new approach to the study of the economic impacts of government R&D, using the ATP as an example of a new type of government program based on a publicprivate partnership. Traditional economic studies find that government funding displaces private investment but most studies have focused on defense procurement contracts. The ATP, by contrast, is a program in which the government is more of a partner than a customer; there are strong incentives to cooperate, share information, and undertake early-stage R&D, and government evaluation certifies that the technology is investment-worthy.

Differentiating Characteristics of the ATP. In assessing the ATP, Dr. Feldman suggested that it is important to understand that the program has the following critical characteristics that differentiate it from other government R&D programs:

  • a focus on developing the economic benefit of early stage, high-risk, enabling innovative civilian technologies;

  • an emphasis on the formation of partnerships and consortia;

  • a rigorous, competitive selection process with an independent evaluation of the project's technical merit, commercial worthiness, and potential for broad-based economic benefits; and

  • debriefings for those who apply but are not selected.

Surveying Winners and Non-winners. To learn about the impacts of these differences on ATP applicants, Dr. Feldman surveyed the 1998 applicants, both winners and non-winners, seeking answers to two questions: “Does the ATP application process reward risky, broad-based research projects?” and “Does the ATP award encourage subsequent investment by private investors?” The survey's findings show that the ATP, as intended, promotes new collaboration among awarded firms and partnerships with universities and government research institutions. She also found that awarded companies express greater willingness to share research results with other firms than do non-winners, which underscores the precompetitive nature of the selected projects. Interestingly, the survey also reveals that firms participating in the selection process find it to be fair and valuable. This view is held by both the award winners (as one might expect) and the majority of the firms who were unsuccessful in their applications (which one would not expect). This speaks well of the selection process.

Micro and Macro Impacts

Mark Ehlen, of the Office of Applied Economics in the NIST Laboratory for Building and Fire Research, presented an ATP-commissioned economic analysis



132 See the paper by Maryann Feldman and Maryellen Kelley, “ Leveraging Research and Development: The Impact of the Advanced Technology Program,” in this volume.

Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×

Page 73

of the micro- and macro-economic impacts of one ATP project: the Flow-Control Machining project carried out by a team led by Extrude Hone Corporation, a small machine tool company. He outlined the joint venture and its structure, discussed the process technologies that were developed, and reviewed the results of the economic analysis of the technologies in their first application in the automotive industry. His study estimated the expected economic impacts of the technology under two scenarios: one involving a brief penetration of the market and the other a longer and more significant penetration. The estimates show that, in either scenario, the vast majority of benefits are captured not by the company doing the R&D, but as spillovers to other firms and customers.

Advancing Manufacturing Technologies

Larry Rhoades, President of Extrude Hone Corporation, the company whose technology was assessed, reviewed his experience of two separate ATP projects, one on Flow Control Machining and another on 3D Printing. Ordinarily the company makes a gradual transition in introducing a new technology, from the complex geometries and difficult materials of the aerospace industry to the high-volume production of the automobile industry, over a period of 10 to 20 years. With the ATP project on Flow Control Machining, there was one giant leap, and the results are already in production throughout the industry. This transition, in fact, might not have occurred had it not been for the ATP.

Mr. Rhoades noted that we are moving beyond the traditional industrial R&D strategies, which begin with manufacturers' efforts to translate customers' desires and government regulations into product designs, and are then given to manufacturing engineers to devise appropriate processes. Markets are moving too fast now for this to occur. The ATP is an example of a new model, which involves designing processes and products together through a process of mutual feedback.

Discussion

Questions from the audience for this panel raised several points. Bernard Gelb of the Congressional Research Service asked Maryann Feldman about her statistical controls in the survey she reported. He observed that the 50 percent response rate is rather high, perhaps suggesting a self-selection bias by those with good things to say about the ATP. Dr. Feldman replied that there is no obvious statistical control for that bias, but the bias could just as easily have been negative, with non-winners complaining, as many in fact did. At the same time, she added, many successful applicants had not replied to the survey.

Bill Long of Business Performance Research Associates suggested to Dr. Feldman that she should consider surveying companies that have not applied in order to strengthen the case that ATP projects fund technologies that would not

Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×

Page 74

be able to attract private capital. Her team had considered that, she said, but had no way to construct such a sample. There are data on companies' funding histories before applying to the ATP, but not on companies that have never applied.

Lewis Branscomb suggested that Feldman's team approach venture capital firms for information on firms that they invest in. She replied that such a retrospective study—relying as it must on people's recollections—would introduce its own biases.

Bill Long asked Larry Rhoades whether it is typical of his industry to capture only 1 percent of the total value created by a new innovation, as documented by Mark Ehlen's study. Mr. Rhoades answered that the industry's ability to harness the value created in a manufacturing process is constrained. The typical company is small in relation to its major customers, so it does not have a strong negotiating position. Second, these companies need dozens of specific applications to make a new process pay for itself, and distributing the costs across these many customers is difficult.

Assessing Progress: Case Study Clusters

David Austin of Resources for the Future introduced this panel, which reviewed the latest assessment activities of the program.

Organ Transplants

David Ayares, Vice President of Research of PPL Therapeutics, announced that his company had received its first grant for the development of xenogenic organ transplants for human patients, which are being developed in genetically altered pigs. His company is a subsidiary of the British firm that made news with Dolly, the cloned sheep. The technology it is developing in the ATP project proved too risky for private investors, and the subsidiary company was on the verge of being closed by the parent company when it discovered and applied successfully to the ATP. Now it is the proud custodian of a litter of five little pigs, for which the ATP can take credit. Venture capitalists have expressed interest. More important, he said, if the technology proves successful, thousands of people's lives will be saved. The ATP has been decisive in creating this opportunity.

Counterfactual Cases

Todd A. Watkins, of Lehigh University, presented his innovative method of using case study methodologies to construct counterfactual cases in estimating the economic impacts of new technologies. In doing so, he is tracking both the economic value to participating firms and the knowledge spillovers to nonparticipating firms. By using “snowballing” interviews with customers, suppliers, and competitors, he and his team are attempting to map the technology diffusion pro-

Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×

Page 75

cess one level deeper. In addition, they have developed a “dynamic” model for the defender technologies, replacing the traditional static model. 133

Tissue Engineering

Tayler Bingham of the Research Triangle Institute reviewed an RTI study of the economic impacts of seven completed ATP projects in medical technologies involving tissue engineering. 134 The study team modeled the entire process, from R&D to applications, to estimate the potential benefits of the technologies. Because health goods are not traded on markets, they developed a measure they called “quality-adjusted life year.” (Briefly, a year of life in full health is assigned a value of 1; death is assigned a value of 0; and a year of life at less than full health is assigned a value between 0 and 1.) The study estimated the expected social return on the public (i.e., ATP) investment in these technologies at $34 billion, in net present value terms, or an internal rate of return (IRR) of 116 percent over the study period. The expected total social return on both public and private investments is estimated at $109 billion, net present value, or an IRR of 115 percent over the study period. The expected total private return on these investments is estimated at $1.6 billion in net present value ($914 million of it attributable to public funding), or an IRR of 12 percent over the study period. 135

Economists in a Row…

The panel discussant was Henry Kelly, Associate Director for Technology, White House Office of Science and Technology Policy. He recalled Harry Truman's perhaps apocryphal dictum: “If you took all of the economists and laid them end to end, they'd point in all directions.” Fortunately, one place they all point in the same direction is in their agreement that the social rate of return to R&D investments is much higher than the private rate of return to R&D investments. 136 It is beyond debate at this point. It is also true that the ATP is unique in the extent to which it has been evaluated, he said. If the program were not in place, addressing the problem of under-investment in commercializing R&D, we would need to invent something like it.



133 Todd A. Watkins and Theodore W. Schlie, “Diffusion Pathways for Advanced Photonics and Optoelectronics Technologies,” paper presented at the Western Economics Association conference, Vancouver, British Columbia, 1 July 2000.


134 S. A. Martin, D. L. Winfield, A. E. Kenyon, J. R. Farris, M. V. Bala, and T. H. Bingham, A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies: Preliminary Applications to Tissue Engineering Projects Funded from 1990 to 1996, prepared by the Research Triangle Institute for the Advanced Technology Program, NIST GCR 97-737, April 1998.

135 See the paper by Tayler H. Bingham, “
Estimating Economic Benefits from ATP Funding of New Medical Technologies,” in this volume.


136 See Edwin Mansfield, Estimating Social and Private Returns from Innovations Based on the Advanced Technology Program: Problems and Opportunities, NIST GCR 99-780, January 1996.

Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×

Page 76

Social Returns

In the stimulating debate of the issues presented by this panel, Lewis Branscomb observed that private investments in R&D, like public investments, produce excessive social returns. What market failures does the ATP address? The R&D “funding gap” (under-investment by the private sector) is the most often mentioned, but there are others. He urged the audience to consult his recent report “Managing Technical Risk.” 137 Rosalie Ruegg replied that the ATP's purpose goes beyond compensating for the “funding gap,” insofar as the program is designed to support projects with larger than average spillovers.

Todd Watkins picked up that point, suggesting that the ATP should try to identify the common factors of projects that have high spillovers so they can be used in the selection process. 138

Assessing theATP Assessment Program: Challenges and Policy Issues

The study director, Dr. Charles Wessner, opened this panel by encouraging a freewheeling discussion of ways to improve the ATP assessment program and, more broadly, the program itself. He first asked the invited speakers to briefly present their views, followed by an open discussion.

A New Environment

John Yochelson of the Council on Competitiveness said that the ATP must continue to improve steadily over time. Organizations either get better or they get worse. They do not stay the same. The program's mission—funding the advance of enabling technologies that would otherwise not be funded—has not changed in the past decade. Mr. Yochelson observed that the environment in which it operates, however, has changed dramatically in several ways:

  • Concerns about the competitiveness of the U.S. economy have shifted from the production of competitive standard products and processes to the production of high-value or unique products and processes. Yochelson said that ATP responds to this shift.

  • The process of innovation is increasingly global. Again, this change plays to the ATP's strengths.



137 L. M. Branscomb, K. P. Morse, and M. J. Roberts, Managing Technical Risk, op. cit. See also L. M. Branscomb and Philip E. Auerswald, Taking Technical Risks, op. cit.

138 Adam Jaffe identified some of the common factors of projects that have high spillovers in his report as well as the difficulties in identifying them in advance. See Adam B. Jaffe, Economic Analysis of Research Spillovers: Implications for the Advanced Technology Program, NIST GCR 97-708, Brandeis University and National Bureau of Economic Research, December 1996.

Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×

Page 77

  • The U.S. private sector has increased its focus on unique and high value products and services. This trend cuts the other way for the ATP, which is intended to compensate for companies' unwillingness to make such investments. How can the ATP address that issue?

  • The U.S. economy is performing much better than it was in 1990, with a greater sense of competitive urgency on the parts of industry and government, and the federal budget is less constrained. These shifts would seem to favor the ATP.

  • Economic power has grown more decentralized, with smaller, more entrepreneurial firms crowding out large firms as engines of growth and political power in the public sector flowing to states and localities. The ATP must take advantage of this change.

Maryann Feldman expressed her satisfaction with the day's discussion, although she suggested that in the future, more representatives of universities and state and local governments might be invited to participate because they are increasingly responsible for promoting technology development. 139

MEP vs. ATP

William B. Bonvillian, Legislative Director and Chief Counsel to Senator Joseph Lieberman, drew the group's attention to the divergent political paths taken by the ATP and MEP. Both programs were founded at the same time and under the same legislation. The MEP developed and spread nationwide quite smoothly, with little meaningful opposition. The ATP's course has been erratic, particularly in terms of funding; today it is probably stable, but we should not be complacent about its future. Can we create a high-quality program, like the ATP, with similarly strong support? The best approach, he suggested, might be to take a page from the MEP's book, and bring governors and other elements of state government into the picture.

David Goldston agreed that the ATP seems to be “stuck” politically. One reason is that much of the program's original rationale—the specter of Japan, for example—is no longer taken seriously. And for many, the ATP represents an inappropriate role for government in helping industry. We need to develop a new rationale for the conditions of today. Otherwise the ATP cannot survive, let alone grow.

Reducing Transaction Costs

Todd A. Watkins of Lehigh University praised the ATP's role in reducing the transaction costs of building partnerships, or social networks. These social


139 For data on state activities see National Science Foundation, State Science and Engineering Profiles and R&D Patterns, NSF 00-329, Arlington, VA: National Science Foundation, 2000.

Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×

Page 78

networks are the basis of what many scholars today call “social capital,” which is thought to promote a strong and flexible social order, but is not provided by markets. This social capital—expressed through collaborative relationships—may offer political advantages to the ATP's supporters. The program might stress this aspect of its mission more publicly, for example, through a variety of means, such as workshops and publications.

Micro, Not Macro, Analysis

Turning to another topic, Dr. Watkins expressed skepticism about the use of macroeconomic analysis in evaluating the ATP. Given the small size of the ATP's budget, no impact on the economy at large can reasonably be calculated. Even in the case of defense—where much effort is spent trying to identify positive spillovers into the economy—it is hard to make such a case. 140

After this provocative series of presentations, the discussion was opened to questions from the audience.

Larry Rhoades said he was troubled by attempts to compare the MEP and the ATP. The two programs are very different. Each is successful in its own terms. To widen support for the ATP, he suggested, the program should work hard to apply the results of its evaluations and publicize its contributions to the economy.

Dr. Wessner asked the group to comment on the extent to which evidence from economic evaluation would be persuasive in political terms.

Sustaining a “Creation Myth”

David Goldston expressed doubt, observing that a basic problem for the program is that it has no sustaining “creation myth” to give it the aura of a national objective. Lewis Branscomb expressed his regret over the loss of the ATP focused programs. He suggested that political support for such an initiative might be modeled on the NSF K–12 education programs, which have been widely supported even though they trespass on a state and local prerogative. 141 The goal, he affirmed, is to attack a well-identified problem with enough resources to make a difference.



140 ATP has used the REMI model (Regional Economic Modeling, Inc.) in several cases after a microeconomic analysis has first been done and it has been determined that project effects can be reflected in REMI's input-output matrix and the project participants comprise a large part of the industry sector. Use of the model allows national effects to be estimated. An example was presented by Mark Ehlen in his case study of the application of a new machining technology in the automotive sector.

141 Branscomb is referring to the NSF's Statewide Systemic Initiatives Program. Founded in 1991, the program makes grants to school districts for reforms aimed at fundamental transformation in pursuit of excellence in mathematics and science education. For an elaboration of this point, see Branscomb and Keller, eds., Investing in Innovation, op. cit., pp. 492-493.

Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×

Page 79

Mr. Bonvillian assured the group that he did not favor measures to water down the technical excellence of the ATP, but that the ATP should find some way to build grassroots support throughout the nation.

The group discussed the extent and the strength of local and sectoral support for the ATP. Maryellen Kelley observed that the ATP at one point had a “focused” (that is, sectoral) approach to some of its grants. In practice, however, the partnerships formed tend to be geographically diffuse.

David Goldston observed that MEP, unlike the ATP, had never suffered from organized and committed opposition, because most policy makers see it as an economic development program, not a science and technology program. The ATP might build wider support if it offered regional workshops or other economic development activities.

John Yochelson, David Goldston, and William Bonvillian discussed the pros and cons of regional and state-based services. A state-based program, Goldston said, would lend itself to political interference, harming the program's merit basis. Yochelson suggested that organizing support through regional associations of states, crossing both state and party boundaries, could build support while limiting the potential for interference. Goldston argued that such an approach would create pressures to spread the program's resources evenly around the country, without regard for technical excellence. Bonvillian expressed doubt that a regional organization would develop enough political support for stability, and he stressed the advantages of a program structure that allowed governors to take credit for successes.

Concluding Remarks Charles Wessner, National Research Council

Dr. Wessner concluded the proceedings with a few observations on the program and its goals. The first is that this meeting had again emphasized the quality and diversity of the ATP assessment effort. Based on several years of reviewing federal partnerships, the ATP assessment program clearly surpasses other U.S. partnership programs with its rigor, scope, and independence. 142

Second, perhaps because of its generic nature, the program seems to have been required to meet a higher standard of evaluation than many mission-oriented programs, which had relied on cooperative research agreements, or significantly larger generic programs such as the $1.2 billion SBIR program. The fact is that the ATP assessments, often conducted by independent economists, many under

142 D. Mowery, Using Cooperative Research and Development Agreements as S&T Indicators: What do We Have and What Would We Like? presentation before National Science Foundation conference, Workshop on Strategic Research Partnerships, 13 October 2000, publication of proceedings pending. See also National Research Council, The Small Business Innovation Research Program: Challenges and Opportunities, op. cit.

Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×

Page 80

contract with the National Bureau of Economic Research, strongly suggest that the program is meeting its goals. Few other federal programs have embraced this level and intensity of assessment and sought to apply its results as diligently as the ATP.

These observations are not meant to suggest that the program cannot be improved. Of course it can be, and the NRC report will make suggestions in that regard. Yet the ongoing evaluation of the program and its political difficulties should not be allowed to obscure what we have learned. There is a growing and increasingly weighty body of evidence indicating that:

  • 1) The program is achieving its technology development goals in the fields of information technology and biotechnology. For example, the assessment program has recorded advances in new processes and procedures for printed wiring boards, the use of gallium arsenide to achieve improvements in integrated circuits, testing and aligning extremely precise coated mirrors, and enhancing data storage capabilities. In the field of biotechnology, it has supported advances in rebuilding lost or damaged human tissues, extracting more information from genetic sequencing of DNA, and developing non-toxic bioabsorbable polymers for repairing bone fractures.
  • 2) Through its assessment program, close management oversight, and industry cost sharing, it has developed an effective self-correction mechanism. The assessment program helps management identify the impact of its awards. Close oversight ensures that projects that are not achieving specified goals can be, and are, cancelled. 143 As importantly, cost sharing ensures that private-sector partners withdraw from unpromising technical development of their own volition. Such mechanisms, and the flexibility they engender, are an essential feature of a modern technology development program. 144

143 See Long, Advanced Technology Program: Performance of Completed Projects—Status Report Number, op. cit. This March 1999 report describes 50 projects of which 38 were completed and 12 were terminated (p. ix). For a discussion of the project achievements, see Chapters 2-8. Appendix B provides a brief description of why the projects were terminated (p. 131).

144 See P. Grindley, D. Mowery, and B. Silverman, “SEMATECH and Collaborative Research: Lessons in the Design of High-Technology Consortia,” op. cit., p. 736.. The consortium's goals changed over time, reflecting the changing perceptions of its members' needs. The authors note that this operational flexibility is a strength and is probably essential in an industry evolving as rapidly as the semiconductor industry. See also R. R. Nelson, Government and Technological Progress, pp. 454-455. The cost-share feature of this program is an important and positive difference between the open-ended procurement failures documented by Cohen and Noll (The Technology Pork Barrel, op. cit.), where incentives for private contractors to withdraw from failing development programs were absent. Cohen and Noll also emphasize the need for flexibility in program management. Ibid., p. vii.

Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×

Page 81

  • 3) The technologies ATP supports are meeting the program goals of
    • a) improved efficiency and competitiveness (e.g., Extrude Hone's contribution to manufacturing efficiency and the environment);
    • b) more rapid commercialization of new welfare-enhancing technologies with positive spillovers (e.g., the General Electric mammography diagnostic instrument); 145 and
    • c) the development of significant new scientific discoveries, such as the PPL Therapeutics organ cloning.

Common Questions

Today's discussion was particularly useful in that it clarified some common questions or misunderstandings about the ATP. For example, questions are often asked about the need the ATP fills for a small company, why a large company would or should participate in a government-supported program, and what need is met by the ATP that is not already met by the well-developed venture capital industry in the United States. To take each in turn, Elizabeth Downing provided valuable insights concerning the challenges faced by a new start-up with a promising but technically challenging technology. Her views on the advantages of an ATP award in terms of administrative simplicity, long duration, and entrepreneurial control are particularly valuable. Her observations are especially relevant because she has had recent experience with a variety of federal programs and finds the ATP an excellent source of early finance for a technology of interest to a variety of federal mission agencies. Her views were complemented by Charles Trimble's observation that the ATP is an effective means to transfer funds, with minimal overhead, directly to researchers. The representative from IBM, Kathleen Kingscott, provided an equally valuable presentation explaining that the rapid pace of technological change and the diffusion of technical expertise means that even a major company with a large and successful internal R&D program finds it necessary to partner with small companies and universities through programs such as the ATP to identify new technologies. She described why IBM finds the ATP an effective vehicle to collaborate on the development of enabling technologies. Perhaps most instructive was the presentation by David Morgenthaler, who described the inherently different objectives and functions of venture capitalists and the ATP and affirmed the need for more government investments in support of promising technologies of the type undertaken by this program.



145 Recently approved for clinical use by the Food and Drug Administration, the new system represents a significant technological advance in breast cancer detection. It uses a unique amorphous silicon detector that provides high-quality imaging that can be digitally enhanced and rapidly verified. A 1995 ATP project awarded to General Electric and EG&G Reticon developed a new manufacturing process that significantly reduced the manufacturing cost of the amorphous-silicon panels used in the new detection system, making this superior detection system more affordable and available to a greater number of women. See http://www.nist.gov/public affairs/update/upd000410.htm#Health

Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×

Page 82

There was also much discussion of the extensive ATP assessment program. The outside assessments, complemented by in-house evaluations, demonstrate that the program is achieving its goals in many cases—though not all—and that the program is willing to terminate projects that are not achieving their goals. 146 As noted, this ability to self-correct in a timely manner is an important feature of the ATP. The willingness to accept the inevitable failures and cancel funding is in itself a valuable aspect of the program and one that is unfortunately only too rare. Government programs that can demonstrate an ability to meet their goals, particularly inherently uncertain R&D goals, are also rare. Programs that can positively contribute to these goals can make contributions to U.S. economic growth and international competitiveness. Indeed, as Professor Feller notes, the extensive ATP assessment program has itself led to significant progress in understanding important aspects of the U.S. innovation system and supported the development of methodologies for its analysis.

Changes Needed

As mentioned, there are clearly issues for the NRC assessment to address. For example, issues such as the timing and speed of the award process should be considered, as well as the possibility of concentrating resources in thematic areas, better integration of assessment results in the decision process, and the need to ensure sufficient program scale for maximum impact. There is the related possibility of the program undertaking more “work for others” as do a number of the DoE laboratories. This approach was suggested by Francis Collins of the National Human Genome Research Institute at the initial NRC meeting to review the ATP. These potential improvements might prove valuable to the new ATP management. Yet as the NRC study goes forward, it is important we note and record what the researchers and program participants—winners and losers alike—are saying. What they seem to be saying, and what the outside research shows, is that this is a federal program meeting its challenging goals.

146 See William F. Long, Advanced Technology Program: Performance of Completed Projects—Status Report Number 1, op. cit.

Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×
Page 63
Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×
Page 64
Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×
Page 65
Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×
Page 66
Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×
Page 67
Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×
Page 68
Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×
Page 69
Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×
Page 70
Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×
Page 71
Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×
Page 72
Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×
Page 73
Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×
Page 74
Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×
Page 75
Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×
Page 76
Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×
Page 77
Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×
Page 78
Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×
Page 79
Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×
Page 80
Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×
Page 81
Suggested Citation:"Summary of Symposium Proceedings." National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Washington, DC: The National Academies Press. doi: 10.17226/10145.
×
Page 82
Next: Findings and Recommendations »
The Advanced Technology Program: Assessing Outcomes Get This Book
×
Buy Paperback | $80.00 Buy Ebook | $64.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

This report examines the operations of the APT, reviews its extensive assessment program, and provides NRC Committee findings concerning the ATP’s operations and recommendations for potential improvements to the program. The report includes a summary of a major conference held in April 2000 as well as seven papers, including surveys of the industry participants or users of the ATP program, a summary of the results of fifty awards, detailed assessments of major joint ventures, and a description of the current selection process. It is the most comprehensive study to date of the program’s origins, operations, achievements, and assessment. Its conclusion: the program works.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!