Panel VI: Observations and Policy Issues

Introduction

Charles Wessner

National Research Council

Dr. Wessner convened the day's final session by saying that the panel offered the opportunity for unstructured discussion about the Advanced Technology Program (ATP). Dr. Wessner opened the discussion by asking management of the ATP about the trade-off between devoting resources to making new ATP grants versus spending time and money on assessing past grants. Dr. Rosalie Ruegg responded by saying that the ATP does not do detailed case studies on each project, but focuses rather on cases of special interest. Through the National Bureau of Economic Research, the ATP also teams with other researchers, such as Josh Lerner from Harvard, as a cost-effective way to obtain high-quality assessment of the program. The vast majority of ATP funds go to grants for companies, with a small portion going to assessment.

ATP Assessment: Looking Back and Looking Ahead

Barry Bozeman

Georgia Institute of Technology

Dr. Bozeman's comments on ATP assessment looked past assessments of other programs, and made observations about ATP assessment in the future.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 111
--> Panel VI: Observations and Policy Issues Introduction Charles Wessner National Research Council Dr. Wessner convened the day's final session by saying that the panel offered the opportunity for unstructured discussion about the Advanced Technology Program (ATP). Dr. Wessner opened the discussion by asking management of the ATP about the trade-off between devoting resources to making new ATP grants versus spending time and money on assessing past grants. Dr. Rosalie Ruegg responded by saying that the ATP does not do detailed case studies on each project, but focuses rather on cases of special interest. Through the National Bureau of Economic Research, the ATP also teams with other researchers, such as Josh Lerner from Harvard, as a cost-effective way to obtain high-quality assessment of the program. The vast majority of ATP funds go to grants for companies, with a small portion going to assessment. ATP Assessment: Looking Back and Looking Ahead Barry Bozeman Georgia Institute of Technology Dr. Bozeman's comments on ATP assessment looked past assessments of other programs, and made observations about ATP assessment in the future.

OCR for page 111
--> Dr. Bozeman recalled an earlier statement about the ATP being the most studied federal program since Head Start. Taking issue with that characterization, Dr. Bozeman said that most social programs are assessed a great deal, probably more so than the ATP. Nonetheless, Dr. Bozeman said that this was a good metaphor for thinking about the challenges facing the ATP. Dr. Bozeman agreed, however, that the ATP has been studied more than any other technology program. The ongoing ATP assessment has benefited from the development of a number of useful assessment tools. Having read many of the studies on the ATP, Dr. Bozeman said he has found most of them to be well-done and valuable contributions to program assessment. In today's environment in which the Government Performance Review Act (GPRA) is so important, this is no small accomplishment. Another metaphor from Head Start is that the best studies show that the Head Start program did not work very well. The more precise the studies were, the smaller the ''trace effects" of Head Start were, that is, the effect of Head Start quickly wore off for the individuals studied. One lesson of Head Start is to learn from evaluation. Because of Head Start evaluation, another program was developed to address diminishing trace effects and this greatly improved the Head Start program. Dr. Bozeman said that he was pleased to see that the ATP was using its evaluation to improve program implementation. Turning to another Head Start metaphor, Dr. Bozeman said that, in looking back to the old Office of Economic Opportunity and other War on Poverty programs, he doubted that many people would recall which one had the best cost/ benefit ratio and the best rate of return to the economy. The program with the best return was not Head Start, but the Job Corps program. Yet in spite of positive assessments, Congress disliked the Job Corps and the program did not have a long life. The lesson for the ATP is that, regardless of whether returns to the economy are estimated at $20 million or $35 billion, these positive evaluations in themselves are not likely to be program sustaining. Dr. Bozeman said that this realization can be quite liberating. It gives program managers a chance to build a clientele for what really works. He sees this already in the ATP; in addition to estimating economic benefits, Dr. Bozeman said that evaluators are paying attention to program instruments that are effective and are highlighting them. One of the most important metaphors from programs such Head Start is that, overall, the War on Poverty was lost. That, at least, was the prevailing perception in the 1970s, said Dr. Bozeman. However, an authoritative study from the University of Arizona found that, taken over a long time horizon, the returns from the War on Poverty were quite substantially positive, especially in light of the modest amounts invested. Dr. Bozeman said that this would be the likely long-term conclusion on the ATP, namely, that its benefits would be shown to be quite positive. In concluding, Dr. Bozeman said that he hoped there would be an opportunity for this kind of long-term analysis of the ATP.

OCR for page 111
--> The ATP and Legislative Flexibility James Turner House Committee on Science Mr. Turner said that, in the day's discussion, "a convincing case . . . has been made that this program [ATP] really has done some good." Clearly, there are companies in existence doing valuable research that would not have existed without the ATP. And there are excellent research outcomes from existing companies or groups of companies that would not have come about without the ATP. Another overarching fact about the ATP is that it exists in a very political environment; evaluation should take that into account. Mr. Turner said that he does not want to be at a symposium 25 years from now in which a researcher says that the ATP, a program canceled 20 years ago, did great things. Mr. Turner would like to convince ATP's critics of ATP's strengths, so that it is not subject to attack the next time a Republican is in the White House. Mr. Turner also responded to several comments during the day that the ATP statute was "fuzzy" and was not prescriptive enough. Mr. Turner said that this was intentional. The House Committee on Science has a long relationship with the National Institute of Standards and Technology (NIST), and the Committee knows that NIST has a tremendous amount of expertise that the Committee lacks. In building a new program such as the ATP, NIST's expertise was seen as something that justified writing flexibility into the statute. Finally, Mr. Turner said that he hoped that the evaluation tools developed for the ATP could be used to assess other technology programs. This was increasingly important in the GPRA environment. Evaluation of technology programs is more important today than in the past, and researchers can do a great service by adapting tools from the ATP to other programs. The ATP and U.S. Technology Policy Richard Nelson Columbia University Reflecting on how the ATP fits into a broader structure of U.S. technology policy, Dr. Nelson said that the ATP plays an important niche role. It is a program that can accomplish valuable things under certain circumstances, but should not be seen as a general-purpose instrument for technology policy. Dr. Nelson identified three dimensions of the ATP that he sees as constituting its niche: The locus of research activities: Research and development (R&D) in the United States takes place in universities, government labs, special-purpose

OCR for page 111
--> organizations such as SEMATECH, and industry. The ATP is clearly oriented toward providing resources for R&D in industry. The role of industry in molding the allocation of R&D resources: For any U.S. technology program, it is important to have business closely involved in setting R&D priorities. This can be done through advisory committees, through special commissions, or by having business pay a portion of the cost of government-sponsored R&D. The ATP uses the latter approach, which is appropriate for many R&D activities. The balance of benefits accruing to the organization performing the research versus the benefits that are distributed widely to the public: The ATP, Dr. Nelson said, funds R&D in which a ''nontrivial portion of research benefits go private." In commenting on the ATP's role in technology policy, Dr. Nelson said that there are many areas in which the ATP is the appropriate instrument for meeting public goals, but he cautioned that there may be areas in which the ATP is not the appropriate instrument. The ATP and Program Management J. C. Spender New York Institute of Technology Dr. Spender said that today's discussion reminded him of Dr. Nelson's book, co-authored with Sidney Winter, An Evolutionary Theory of Economic Change, in which Nelson and Winter highlighted the search for organizational routines to meet certain goals. In talking about the ATP, Dr. Spender observed that much of the focus on "program instrumentalities" was really about the search for the appropriate routines to manage the ATP. There were three elements of this search: Determination of policy: Are the routines that have been developed in the ATP over several years appropriate to all programs, or just to ATP? Constraints: Are the practices developed in the ATP effectively constrained, through things such as requiring matching funds or encouraging joint ventures? Do these constraints affect the measures of program impact? Achieving objectives: Is the ATP about spillovers? Or are there other objectives? The ATP's objectives remain ambiguous, even after the day's discussions, and Dr. Spender recommended considering the interaction of the public and the private to think this issue through. The history of Silicon Valley, Dr. Spender

OCR for page 111
--> argued, is about the privatization of public goods, goods that were developed using defense contracts. The ATP, in contrast, seemed to encourage the "publicization" of private R&D expertise. The interplay between public and private is the heart of the ATP, and Dr. Spender argued that, to understand this interplay, economic data need to be supplemented with sociological and anthropological data. Case studies could address these issues, but the question then becomes how to interpret case studies. One lens through which case studies could be viewed is the notion of "communities of practice." An industry or a firm may be seen as a community of practice and the ATP is a mechanism that affects this community. Another term for a community of practice might be the innovation system. There must be greater effort devoted to modeling these complex innovation systems, recognizing the interdisciplinary nature of the task. The ATP and Spillovers Christopher Hill George Mason University Dr. Hill of George Mason University observed that the Senate staff most closely involved in the creation of the ATP did not view it as a solution to all of America's economic ills. Rather, the ATP was seen as an experiment, an opportunity to apply some policy prescriptions that had been talked about for years. Conceived as a program modest in size, it was originally intended to focus on computers, biotechnology, and robotics. A clause was inserted into the legislation to provide an "open window" for the program to address subject areas beyond those three. This "open window" has allowed the ATP to pursue exciting new areas and respond to changing economic and technological challenges. In reflecting on the modest scope of the ATP, Dr. Hill said that he contributed a paper to Lewis Branscomb's book, Investing in Innovation, that spent a great deal of time considering other possible technology policy instruments in the event that the ATP proved insufficient. Dr. Hill therefore agreed with Dr. Nelson that the ATP should be seen as one part of a broader technology policy. Dr. Hill then proceeded to try to reframe the externalities and spillover issue. Taking the National Science Foundation (NSF) as an example, Dr. Hill said that nearly everyone is comfortable with what that agency does. As a grant-giving organization, NSF provides a mixture of public and private goods. Faculty members and students at universities receive NSF funds, and this creates goods that recipients can capture privately. Faculty members generate research, which will reflect positively on them when their performance is reviewed, and graduate students receive credentials and training for the job market. Because he received NSF funding as a graduate student, Dr. Hill has been able to enjoy a higher level

OCR for page 111
--> of income than he would have otherwise. Dr. Hill also pointed out that NSF induced him to participate in its program at a low cost; in 1968, when he was in graduate school, his NSF stipend of $213 per month was enough to support him in school for four years. However modest the stipend, it accomplished a public purpose, namely to create a more scientifically and technically trained workforce. One can argue, that the founders of NSF really did not care about his personal success or that of other award recipients. However, he and others were instruments of public policy, not the ends of public policy. As long as he and others like him created appropriate knowledge spillovers, their individual success was irrelevant to NSF. Dr. Hill argued that the ATP is in an analogous situation. It is trying to induce private companies to engage in behavior that has a public purpose. If it were only for private purposes, in his view there would be no justification for the ATP. The ATP tries to do this by giving firms the minimum amount of money to induce them to engage in R&D that they otherwise would not undertake. It does not matter, argued Dr. Hill, whether the firms succeed or not. From the perspective of the program's goals, it does not matter whether awardees succeed or fail. All the ATP wants the companies to do is develop new knowledge and new technologies. To carry that out, it is helpful for those companies to succeed and continue to exist. Moreover, he added that there is the political consideration that "it is not good politics to fund companies and then have them go belly up." The real goal, then, is to generate the spillovers to society, not create profits for private firms. For these reasons, Dr. Hill said that ATP evaluation should pay little attention to its awardees client companies and instead should focus on the spillovers that do not accrue to the awardees. Questions from the Audience Todd Watkins from Lehigh University said that one of the benefits of programs such as the ATP was the social aspect of communities of practice, or what economists call "social capital." These are the benefits associated with collaborative activities and people in industries being able to work with one another easily. Dr. Watkins asked whether assessment of the ATP considered social capital or communities of practice not only among successful ATP firms, but among those that have not won awards. Dr. Watkins noted that the mere process of coming together to apply for an ATP award may yield significant benefits in terms of social capital. In response, Dr. Maryellen Kelley of NIST said that ATP assessment does try to include the social capital benefits of the program. Dr. Wessner observed that many venture capitalists say that there are many more attractive business ideas to be funded than there is venture capital available to fund them. Dr. Wessner asked NIST Director Ray Kammer if that was the case with this program. In response, Mr. Kammer said that ATP funds approximately 1 out of every 10 proposals it receives. If it had unlimited funds, the ATP would

OCR for page 111
--> like to fund between 2 and 3 out of every 10. At the present level, the ATP funds about 10 percent of its applicants, and somewhat more than 20 percent of applications are worthy of funds. Mr. Kammer added that the number of applications is also a function of how much money the ATP receives in its appropriation. When the program budget is expanded, it tends to receive a greater number of applications. Dr. William Long expressed disagreement with Dr. Hill's statement about whether we should care about the health of companies funded by the ATP. Dr. Long said that if an ATP grant recipient does not produce a good or service, it is difficult for the ATP-funded technology to be disseminated in the economy. In response, Dr. Hill clarified by saying that ATP companies do not need to do better than their counterparts in a sector in order for spillovers to be created. Dr. Long added that it is difficult to have spillover effects if there are no products in the market. Concluding Remarks In concluding the panel, Dr. Wessner recalled the introductory remarks of Bill Spencer, noting that the ATP, like other government-industry partnerships, can be seen as an experiment. These experiments have a long history in the United States, dating at least to congressional funding for the telegraph in the mid-nineteenth century. He added that it seems that we do more experimenting in government-industry partnerships in the United States than we are politically or ideologically willing to admit. Dr. Wessner also observed that there was little discussion about activities of other countries in programs such as the ATP. As Dr. Spencer had reminded us at the outset, in other countries, there is little debate over whether government should support industry in technology development; many countries implement substantial programs as a matter of course. As an example, he cited a recent conversation with an American venture capitalist who was concerned that the German government's growing support for its biotechnology industry in the form of very patient capital could pose a challenge to the leading position of U.S. industry. Finally, Dr. Wessner added that although foreign partnership programs face less scrutiny, they perhaps do not benefit from the rigorous assessments that characterize some U.S. programs. With respect to the assessment of the ATP, Dr. Wessner said that, not every agency would invite this type of open discussion with program advocates and opponents, program users, and program researchers. The fact that this type of meeting was sought is much to the credit of the NIST management. The fact that the senior management from the ATP and NIST stayed for the entire event underscored their interest in hearing outside views on the operation of the program. It is to be hoped that more agencies would invite this type of objective assessment. In closing, Dr. Wessner reminded the audience that government support to

OCR for page 111
--> industry, and questions about the appropriateness of that support, have a long history in the United States dating back to the origins of the Republic. Alexander Hamilton's 1791 Report on Manufacturers had stressed the importance of developing new forms of manufacturing and the government's positive role. As a strong proponent of government support for new industries, Hamilton had affirmed that "there is no purpose to which public money can be more beneficially applied than to the acquisition of a new and useful branch of industry, no consideration more valuable than a permanent addition to the stock of productive labor." With this early federalist observation in mind, Dr. Wessner thanked the participants for bringing their expertise and experience to this initial review of the Advanced Technology Program. He added that the Board expected this symposium to contribute both to our understanding of the ATP and importantly, to the Board's overall review of government-industry partnerships. It is anticipated that, in cooperation with NIST, additional analysis of the ATP will be carried out under the auspices of the Government-Industry Partnerships project. In this second phase, we will draw both on the ATP's well-developed program of assessment and outside researchers, as well as the experiences of award recipients, to identify program strengths and potential improvements. Once this additional analysis is completed, the project steering committee expects to develop specific findings and recommendations for the Advanced Technology Program. These will be reported separately and will also be an important contribution to the Academies' broader review of government-industry partnerships.