Assessing the ATP Assessment Program: Challenges and Policy Issues
National Research Council
Charles Wessner opened the panel session by encouraging a free discussion of ways to improve the ATP assessment program, and, more broadly, ways to improve the program itself. These comments will be valuable to the NRC review of the ATP. To open the discussion, he asked John Yochelson, President of the Council on Competitiveness, to offer his views.
Council on Competitiveness
Mr. Yochelson said that ATP must be improved steadily over time. Organizations either get better or they get worse. They do not stay the some. The program's mission—to funding the advance of enabling technologies that would otherwise not be funded—has not changed in the past decade. The environment in which it operates, however, has changed substantially in several ways.
A Change in Focus
First, the focus of concerns about competitiveness of the U.S. economy has shifted. Today the goal is to produce high-value or unique products and pro-
cesses. When the ATP was founded, the problem was seen as an inability to produce competitive standard products and processes. The ATP responds to that change in the policy environment.
Globalization of Innovation
A second, related, trend is that the process of innovation is increasingly global, and its pace is accelerating. Companies turn knowledge into high-value products and processes at facilities throughout the world, sometimes on a twenty-four hour schedule. Furthermore, other countries are getting better at this kind of innovation. Again, this change plays to the ATP's strengths.
Changing Investment Patterns
Third, the private sector in the United States—especially in the past four or five years—has increased its focus on developing unique and high value products and services. This trend cuts the other way for the ATP, which is intended to compensate for companies' unwillingness to make such risky investments. As the federal share of R&D has declined, and private investment has moved into these areas. How can the ATP address that issue?
A Better U.S. Economy
Fourth, the U.S. economy is performing much better than it was in 1990. The sense of competitive urgency, on the parts of both industry and government, has improved. The federal budget is less constrained. These shifts would seem to favor continued support of the ATP.
Finally, economic power has grown more decentralized. In the private sector, smaller, more agile, and more entrepreneurial firms have crowded out large firms as the engines of growth. In the public sector, much political power has flowed to states and localities. ATP must take advantage of this change.
Johns Hopkins University
Adding Other Stakeholders
The day's program, Dr. Feldman said, was a heartening display of the rigor and seriousness with which ATP and its partners in industry have taken their responsibilities for the program. In particular, viewpoints of large and small com-
panies had been well explored. She suggested that in future more university representatives be invited to participate. State and local governments also should be involved, since they are increasingly responsible for promoting technology development and innovation within their borders.
A New Program with a New Paradigm
ATP, she added, represents a paradigm shift in the economics of public R&D funding. Traditionally economists have looked at government programs as procurement-type programs, which could be analyzed in the aggregate terms of displacement effects and the like. But the ATP offers a new way of partnering and leveraging public and private resources. It is critical that economists who do assessments understand the implications of this change.
Office of Senator Joseph Lieberman
ATP vs. MEP
Mr. Bonvillian drew the group's attention to the divergent political paths taken by the ATP and MEP. Although founded at the same time, under the same legislation, MEP developed and spread nationwide quite smoothly, with little meaningful opposition. The ATP's course has been erratic, particularly in terms of its funding; today it is probably stable, but we should not be complacent about its future.
The Need for a Stable Political Model
If the ATP is truly a paradigm shift—if it is truly a way out of the Valley of Death in which America's small firms found themselves a few years ago—then we should seek a new and more stable political model for it. MEP is a program that was created for governors and states. It has a built-in support group in every statehouse and governor's mansion. Governors are proud of their MEP outreach programs. MEP is also popular with industry. As a result, it benefits from an ongoing lobbying network centered in the states and extending down to the grassroots of hundreds of thousands of small manufacturers.
Substantive Success and Political Instability
The ATP is a wonderful success in technical terms. But the designers of the program may not have created the right political model for it. The political model was based on substantive considerations. We wanted to discourage repetitive participation by companies, as in the SBIR. We wanted to focus on smaller and mid
sized firms. We have been successful in meeting most of these goals. But we missed the chance to create a strong political base.
Congress sometimes creates poor quality programs with strong political support. Can we create a high-quality program, like the ATP, with similarly strong support? The trick is to take a page from MEP's book and bring the governors and states into the picture.
Modern economic development policy, after all, stresses the development of regional clusters of industry. The ATP contributes to that strand of policy and should be given political support that corresponds to its substantive excellence.
Office of Congressman Sherwood Boehlert
The Need for a New Rationale
David Goldston agreed that the ATP seems to be stuck politically. In Congress the controversy over its funding has assumed the character of an annual ritual. One reason is that much of the program's original rationale—the specter of a hypercompetitive Japan, for example—is no longer taken seriously. The ATP today is, for many, a political symbol of an inappropriate role for government in helping industry. Accordingly, we need to develop a new rationale which better reflects the conditions of today. The ATP cannot survive, let alone grow, if it does not bring itself up to date in this important respect.
Economic vs. Technology Development
Bill Bonvillian's proposal for a nationwide network of political supporters at the grassroots would require a radical rethinking of the program. Congress traditionally has distinguished between economic development programs and science and technology programs. In the former (characterized by MEP) states are active, and political lobbying is expected. Science and technology programs (DARPA's, for example) are expected to be governed by impartial merit review and immune from political considerations. The most conspicuous programs that bridge these two models generally involve pork barrel spending for universities, supported by states. Usually these projects are widely deplored.
The ATP assessment process has had an important impact on the political environment, by making it hard to attack as simply poorly run. One no longer hears this argument much. But, as Irwin Feller pointed out this morning, these assessments must be viewed primarily as contributions to the way the program is run, and not as a panacea that will eliminate the political debate. That debate dates back 200 years, and one can always formulate legitimate questions that cannot be satisfactorily answered by the output of an economic model.
Economists, beyond their role as evaluators, need to learn how to help answer the vital questions about ATP. What are the real gaps and unseen failings in the current economy, and is the ATP designed to address them? We must be prepared to debate those questions? The existing dependable battle lines cannot be depended on for the friction that keeps the program going forever.
Todd A. Watkins
From the Hill's perspective, Dr. Watkins said, the search for political traction for the program depends on the ability of the ATP to build relationships. As Kathleen Kingscott told us this morning, even a company as big and global as IBM finds the ATP valuable as a source of new relationships with R&D partners and suppliers. Elizabeth Downing, CEO of a tiny new company, finds the ATP useful as a kind of marriage broker—a source of new contacts.
Building “Social Capital”
In this role, the ATP reduces what economists call the transaction costs of building partnerships or social networks. These social networks are the basis of what many scholars today call “social capital,” which is thought to promote a strong and flexible social order but is not provided by markets. This social capital—expressed through collaborative relationships—may offer political advantages to the ATP's supporters. The program might stress this aspect of its mission more publicly, for example, through a variety of means, such as workshops and publications.
This lesson is one that can be drawn from the European equivalent of the ATP, the Framework technology program (originally called Esprit). This program predates the ATP by several years. 28 It was based on a similar market-failure argument. As the Europeans gained experience with the program, it began to stress the social capital contributions of the program, which in Europe has great political salience.
One should be skeptical about applying macroeconomic analysis to the ATP. Plugging microeconomic data, such as dollar cost savings of individual compa-
28 For a description of the Fifth Framework Programme (1998) see Jorma Routti and William Cannell, “European Union Research Programs,” in National Research Council, New Vistas in Transatlantic Science and Technology Cooperation, Charles W. Wessner, editor, Washington, D.C.: National Academy Press, 1999, p. 103. This program favors “partnerships and networks of research actors, public and private, which are more strongly oriented towards utilization and uptake of results” (p. 103). “The majority of the funding to date has been allocated to five broad themes: energy, life sciences, environment, industrial and materials technologies, and information and communications technologies” (p. 105). Funding for the Fifth Framework is 14.96 billion Euros over five years. http://europa.eu.int/comm/research/growth/leaflets/en/whatisthe5th.html.
nies, into macroeconomic models of the economy is a meaningless exercise. The ATP's budget of a few hundred million is “in the noise” so far as the national economy goes. Even in the case of defense—where much effort is spent trying to identify positive spillovers into the economy—it is hard to make such a case. The ATP should spend its money on the microeconomic studies.
QUESTIONS FROM THE AUDIENCE
Larry Rhoades said that he found attempts to compare MEP and the ATP troubling. The two programs have vastly different focuses. MEP (on whose National Advisory Board he sits) is devoted to keeping small manufacturers relatively up to date by addressing their rather well-defined and common problems; it therefore is appropriately handled regionally. The ATP, on the other hand, seeks original solutions to unique problems, which cannot be constrained by state borders. The ATP could do two things to improve itself:
spend even more effort on evaluating the program and tightening its management, to build support for the idea that the program is a revenue enhancer, and not an expenditure program; and
build relationships between product and process developers.
Charles Wessner asked the group to comment on the extent to which evidence of positive spillover to the economy would be politically persuasive to policy makers.
Only to a limited extent, said Mr. Goldston. Many science and technology programs have “creation myths” that give them the aura of solid and sustained national contributions, whether earned or not. DARPA and NSF, for example, are credited with the creation of the Internet, and are supported for that reason. NIH, similarly, is given credit for all health advances. The ATP does not yet have such a reputation for national achievement.
Reviewing the extent of local and sectoral support for the ATP, it was observed that the ATP at one point had “focused” competitions in particular sectors. Additionally, ATP partnerships often spread quickly beyond the locality of the original grant performance.
Mr. Bonvillian emphasized that the ATP and MEP are very different programs serving different constituencies. The ATP must retain its focus on high-risk technology. But the intense and broad political support for MEP is an important feature of that program. The ATP has no structure for involving business, universities, and local and state officials. Every governor has a sense that new technology is an important driver of growth, but ironically there is no grassroots
support in the states. He added that this may also be an issue of program scale. The ATP may lack the critical mass to develop widespread support.
An Illegitimate Role?
David Goldston responded that MEP has never had organized and committed political opposition. Its activities—economic development through industrial extension—are nearly universally accepted as a government mission. The ATP, on the other hand, has always attracted the attention of some in Congress and elsewhere who regard commercial technology development as a fundamentally illegitimate role for the federal government. The ATP might find wider support if it offered, separately from the grant-making apparatus, regional services such as clearinghouses for technical information, on the model of MEP.
A Regional Approach
John Yochelson asked whether a regional approach might mitigate some of the disadvantages of the state-based approach. Regional associations of states are quite common for dealing with problems that extend beyond state borders. If such an organization crossed party lines and offered a way for states to cooperate, it might prove popular with governors and state business associations, whose support for the ATP is now lukewarm.
David Goldston pointed out that such an approach would solve some problems, but produce others. There would be great pressure to spread resources evenly around the country, with one such center in every region, without regard for technical excellence. Such rhetorical strategies may be useful in the short term. For example, the university pork barrel movement began with the public recognition of universities as engines of economic growth, and quickly took on a life of its own, creating pressure to distribute funds and programs nationwide.
William Bonvillian said he doubted that a regional organization would promote enough political horsepower to build stable support. He would oppose efforts to reduce the merit basis of ATP grants. Still, a program structure that allowed more governors to take credit for successes would clearly bring advantages.
Referring back to comments on the globalization of R&D, Charles Wessner observed that the term “globalization” is often used loosely. In fact, products are assembled and marketed globally, but technology tends to be developed in local concentrations. The American economy's ability to spring back from competitive challenges may be due in part to this local concentration of specialized skills, the deep U.S. capital markets, and the relatively lower penalties for failure among U.S. entrepreneurs. When special effort is required, public alarms are raised over
specific competitive challenges. In contrast to that of most industrial nations, U.S. policy rarely gives the same level of sustained attention to the nation's technology and manufacturing base. 29
A Need for Structured Programs
Lewis Branscomb regretted the loss of the ATP focused programs. NSF's multibillion-dollar initiative in K–12 education—not traditionally a federal concern—achieved support, he pointed out, because it addressed an undisputed problem by applying the best available expertise. Similarly, he suggested, the ATP could assemble a series of sectoral programs to identify competitive problems with technological implications. These programs could build support through partnerships of industry, labor, universities, and state and local government, committing them to act on business problems, with the federal government offering an injection of technological expertise. Such a program would have the advantage of being big enough to be visible on a national scale and focused enough on local problems to generate committed and bipartisan support.
Jean Powell of the ATP assessment staff responded to Todd Watkins's suggestion that macroeconomic impact studies be de-emphasized by ATP in favor of microeconomic studies. In general, she said the ATP management agrees and does not do much macroeconomic study of its spillovers. The main exception, in fact, is the automotive project reported on earlier by Mark Ehlen and Larry Rhoades.
An Experiment in Selection
Elizabeth Downing said that the forces arrayed against the ATP have argued that the private sector would step in, absent the ATP, to fund these long-term investments in enabling technologies at these early stages. As a scientist, she proposed an experiment, in which the selection process would be changed to include representatives of the financial community. A panel of venture capitalists could be allowed to review applications, and given access to any of the technologies they found promising. That would settle the question of whether the private sector can serve this function.
John Yochelson asked whether a representative group of venture capitalists might be willing to testify publicly or privately to Congress that ATP does contribute to meeting the “funding gap.”
29 There are U.S. programs focused on general improvements in manufacturing, such as the Manufacturing Extension Partnership Program, and programs in the National Science Foundation Engineering Research Centers. There are also major programs focused on specific sectors such as the Partnership for the Next Generation Vehicle or the Advanced Battery Consortium.
Does ATP Generate Jobs on a Regional Basis?
Joel Yudken, AFL-CIO, warned that the group may be missing the political forest for the trees. The comparison with MEP may be misleading, since it has a very different mission. The ATP clearly addresses a market failure. Labor, for its part, will not lobby for the ATP unless there is a clear understanding of how it creates jobs and how it affects workers. Organized labor understands that new technology improves productivity and leads to growing incomes for workers. But it will need some clear evidence that ATP projects will pay off in these ways in the industrial sectors where unions are strong. Mark Ehlen's study is very persuasive in that respect and more use should be made of it in making this point. The likely regional distributions of new jobs is of particular interest. If the ATP is to prosper, it must show that it has a real payoff for real people, not just venture capitalists.
About 60 percent of ATP applicants, according to Maryann Feldman's study, are concentrated in four states, said Bill Jones. That result—which reflects the concentration of high-technology industry—is at the root of the narrow political support for the ATP. But perhaps a broader base can be built on the beneficiaries of these technologies, from organ transplant recipients to the drivers of automobiles.
THE ATP DIRECTOR'S THANKS
Alan Balutis, the new Director of the ATP, expressed appreciation of the impartial and penetrating analysis to be expected from the STEP Board's study. Rosalie Ruegg, he added, has built the ATP's assessment program into one that is unique in government. Her upcoming retirement is one of the few bleak spots in his outlook as the program's new director.
National Research Council
After thanking the participants, Dr. Wessner concluded the proceedings with a few observations on the program and its goals. The first is that this meeting had again emphasized the quality and diversity of the ATP assessment effort. Based on several years of reviewing federal partnerships, the ATP assessment program clearly surpasses other U.S. partnership programs with its rigor, scope, and independence. 30
Second, perhaps because of its generic nature, the program seems to have been required to meet a higher standard of evaluation than many mission-oriented programs, which have relied on cooperative research agreements (CRADAs), or significantly larger generic programs such as the $1.2 billion SBIR program. The fact is that the ATP assessment, largely conducted by independent economists, many under contract with the National Bureau of Economic Research, strongly suggests the program is meeting its goals. Few other federal programs have embraced this level and intensity of assessment and sought to apply its results as diligently as the ATP.
These observations are not meant to suggest that the program cannot be improved. Of course it can be, and the NRC report will no doubt make suggestions in that regard. Yet the ongoing evaluation of the program and its political difficul
30 D. Mowery emphasizes this point in “Using Cooperative Research and Development Agreements as S&T Indicators: What do We Have and What Would We Like?” presentation before the National Science Foundation conference, Workshop on Strategic Research Partnerships, 13 October 2000, publication of Proceedings pending. See also National Research Council, The Small Business Innovation Research Program: Challenges and Opportunities, Charles W. Wessner, editor, Washington, D.C.: National Academy Press, 1999.
ties should not be allowed to obscure what we have learned. There is a growing and increasingly weighty body of evidence indicating that:
- 1. The program is achieving its technology development goals in fields such as information technology and biotechnology. For example, the assessment program has recorded genuine advances in new processes and procedures for printed wiring boards, in the use of gallium arsenide to achieve improvements in integrated circuits, in testing and aligning extremely precise coated mirrors, and for enhancing data storage capabilities. In the field of biotechnology, it has supported advances in rebuilding lost or damaged human tissues, extracting more information from genetic sequencing of DNA, and in the development of bio-absorbable polymers for repairing bone fractures. 31
- 2. Through its assessment program, close management oversight, and industry cost-sharing, it has developed an effective self-correction mechanism. Such mechanisms, and the flexibility they engender, are an essential feature of a modern technology development program. 32
3. The technologies the ATP supports are meeting the program goals of
improved efficiency and competitiveness (e.g., Extrude Hone's contribution to manufacturing efficiency and the environment);
more rapid commercialization of new welfare-enhancing technologies with positive spillovers (e.g., the GE mammography diagnostic instrument); 33 and
development of significant new scientific discoveries, such as the PPL Therapeutics organ cloning.
32 See P. Grindley, D. Mowery, and B. Silverman, “SEMATECH and Collaborative Research: Lessons in the Design of High-Technology Consortia,” Journal of Policy Analysis and Management, 13(4):736, 1996. The consortium's goals changed over time, reflecting the changing perceptions of its members' needs. The authors note that this operational flexibility is a strength and probably essential in an industry evolving as rapidly as the semiconductor industry. See also R. R. Nelson, Government and Technological Progress, New York: Pergamon, pp. 454-455, 1982.
33 Recently approved for clinical use by the Food and Drug Administration, the new system represents a significant technological advance in breast cancer detection. It uses a unique amorphous silicon detector that provides high-quality imaging which can be digitally enhanced and rapidly verified. A 1995 ATP project awarded to General Electric and EG&G Reticon developed a new manufacturing process that significantly reduced the manufacturing cost of the amorphous-silicon panels used in the new detection system, making this superior detection system more affordable and available to a greater number of women. See http://www.nist.gov/public affairs/update/upd000410.htm#Health
Today's discussion was particularly useful in that it clarified some common questions or misunderstandings about the ATP. For example, questions are often asked about the need the ATP fills for a small company, why a large company would or should participate in a government-supported program, and what need is met by the ATP that is not already met by the well-developed venture capital industry in the United States.
To take each question in turn, Elizabeth Downing provided insights concerning the challenges faced by a new start-up with a promising but technically challenging technology. Her views on the advantages of an ATP award are particularly valuable, perhaps unique, insofar as she has had direct experience with a variety of federal programs. The representative from IBM, Kathleen Kingscott, provided an equally valuable presentation explaining why even a major company with a large and successful internal R&D program finds it necessary to partner and finds the ATP an effective vehicle to collaborate on the development of enabling technologies. Perhaps most constructive was the presentation by David Morgenthaler, who not only outlined why venture capital and the ATP have inherently different objectives and functions, but also affirmed the need for more government investments of the type undertaken by this program.
Although several speakers suggested the scale of the program should be increased, the program's level of activity is significant. Over the last ten years, it has funded over 460 projects involving over a thousand participants in 157 joint ventures. These awards have accounted for $3 billion in funding for new technologies, with about half of the funds from industry, the other half from the ATP. Interestingly, small businesses play a dominant role in the program. The majority of the ATP awards, currently 66 percent, are made to small businesses, with about 25 percent going to small start-ups. The program has also involved a large number of universities and national laboratories. 34
There was also much discussion of the extensive ATP assessment program. Importantly, these outside assessments, complemented by in-house evaluations, demonstrate that the program is achieving its goals in many cases—though not all—and terminating projects that are not achieving their goals. As noted, this ability to self-correct in a timely manner is an important feature of the ATP. The willingness to accept the inevitable failures and cancel funding is in itself a valuable aspect of the program and one which is unfortunately only too rare.
Government programs that can demonstrate an ability to meet their goals, particularly inherently uncertain R&D goals, represent policy success with potentially large benefits to national welfare, the accomplishment of government
34 As noted above, through the end of 2000, the ATP funded 522 projects with 1,162 participants and an approximately equal number of subcontractors. Through that time, 176 universities had participated as well as more than 20 national laboratories.
missions, and the competitiveness of U.S. industry. Programs that can positively contribute to these goals can make important contributions to economic growth, national competitiveness, and human welfare. Indeed, as Professor Feller notes, the extensive ATP assessment program has led to significant progress in understanding important aspects of the U.S. innovation system and has supported the development of cutting edge methodologies for its analysis.
As mentioned, there are clearly issues for the NRC assessment to address. For example, we should probably consider issues such as the timing and speed of the award process, the possibility of concentrating resources in thematic areas, better integration of assessment results in the decision process, and the need to ensure sufficient program scale for maximum impact. There is the related possibility of the program undertaking more “work for others” as do a number of the DoE laboratories, particularly with regard to tools and medical equipment. This was in fact suggested by Francis Collins of the National Human Genome Research Institute at the initial NRC meeting to review the ATP. Yet as the NRC study goes forward, it is important we note and record what the researchers and program participants—winners and losers alike—are saying. What they seem to be saying, and what the outside research shows, is that this is a federal program meeting its challenging goals.