Government’s Evolving Role in Supporting Corporate R&D in the United States: Theory, Practice, and Results in the Advanced Technology Program
National Institute of Standards and Technology
National Institute of Standards and Technology
The Advanced Technology Program (ATP) supports early-stage technology development efforts by U.S. companies.1 The ATP provides funding support to high-risk R&D projects that have potential for broad-based economic benefits for the nation. The rationale for government support of R&D rests on theory and evidence that the social benefits of R&D are greater than the private returns. The role of ATP as a public-private partnering program extends from providing critical funding to early-stage technology projects, and also includes aspects of encouraging collaboration among firms and other organizations, fostering information exchange, and facilitating technology entrepreneurship activities. ATP’s multifaceted evaluation program provides evidence that ATP is meeting its mission to support high-risk and innovative research by U.S. companies.
THE ADVANCED TECHNOLOGY PROGRAM—OVERVIEW AND IMPLEMENTATION
The Advanced Technology Program (ATP) at the National Institute of Standards and Technology (NIST) was established by the Omnibus Trade and Competitiveness Act of 1988, with the mission of supporting U.S. companies in pursuing early-stage, high-risk research to develop new technologies that have great potential for producing broad-based national economic benefit. The ATP funds industry-led research and development (R&D) projects that have high technical risk and inventiveness and promising potential for broad economic impact.
ATP began in 1990 in response to the erosion of U.S industry’s international competitiveness in strategic markets and the relative slowness of U.S. firms in translating inventions created in universities, national laboratories, and corporate laboratories into innovative products and processes (National Research Council 1999; Ruegg and Feller 2003). ATP provides cost-shared funding to industry to accelerate the development and broad dissemination of challenging, high-risk technologies that promise significant commercial payoffs and widespread benefits for the nation. This unique government-industry partnership helps companies accelerate the development of emerging or enabling technologies. Those technologies, in turn, lead to revolutionary new products and to new industrial processes and services that can compete in rapidly changing world markets.
By assisting in the funding of early-stage technology development, ATP helps propel promising technologies from invention to innovation—that is, make the transition from the laboratory to the marketplace, from demonstration of technical “proof of concept” to commercial introduction of a new technology product or service in the marketplace. During the process of early-stage technology development, when technical feasibility and economic viability are yet to be proven, great risk and fundamental uncertainty characterize the endeavor, and in this context, funding for R&D is often unavailable.
In a study commissioned by ATP, this transition from invention to innovation has been described as a “Darwinian Sea,” where good technical ideas may not emerge from the laboratory due to the inability to find funding (see Figure 1). In the Darwinian Sea, success depends critically on the availability of funding, timely information and contacts, and entrepreneurial ability. For startup technology innovators, the primary sources of support are “angel” investors (wealthy individuals with experience in starting up new companies), venture capital firms specialized in early-stage or “seed” investments, and state and federal government programs aimed at supporting technology and innovation.
ATP challenges industry to take on projects that have higher technical risk, and commensurately higher potential payoffs for the nation, than they otherwise would pursue. The ATP project selection criteria reflect this philosophy. Half of the criteria are based on scientific and technological merit and include an explanation of the innovation, a detailed research plan, and justification that the approach is feasible and has the potential to overcome the technical hurdles. The other half of the criteria are based on the potential for broad-based economic benefits, including benefits to the economy and society that would result from developing the new technology, justification for the need for ATP funding, and a plan for how the technology, once developed, will be commercialized. Proposals that are submitted in response to ATP’s announced competitions are peer-reviewed against the published selection criteria. On average, one out of eight proposals met ATP’s criteria for funding.
ATP accepts applications from single companies and joint ventures. For-profit companies may apply as single applicants to receive an award for up to $2 million over three years to cover project costs. Single-company applicants are required to cover their indirect costs; this requirement encourages the participation of small firms that have low overhead costs. In fact, small businesses (those employing fewer than 500 workers) are thriving in the program (nearly half of all ATP-funded small firms have fewer than 20 employees) and lead two out of three of all projects.2 Single-company applicants often bring in subcontractors (universities or other companies) to participate in the project. Large Fortune 500 companies applying as single-company applicants must cover at least 60 percent of total project costs; this requirement encourages large firms to formally collaborate with others and apply as a joint venture.
At least two separately owned for-profit companies may apply as a joint venture, with both companies substantially contributing to the research effort and to the requirement to cover at least half of the total project costs. Additional organizations (universities, nonprofits, or other companies) may join the joint venture either as formal participants or as subcontractors. Joint ventures can receive ATP funding for up to five years, with no funding limitation other than the announced availability of funds and the organizations’ ability to cover half the total project costs.
ATP announces competitions through the Federal Register and held 44 competitions between 1990 and September 2004. ATP has provided $2.2 billion in awards, and industry has provided an additional $2.1 billion as cost share, for a total of $4.3 billion for high-risk research. Of the 768 projects awarded to date, 550 are to single-company applicants and 218 are to joint ventures. More than 165 universities and 30 national laboratories have participated in ATP projects, reflecting the collaborative nature and the diversity of the projects’ participants. Projects that are awarded are organized into four broad technology areas: advanced materials and chemistry, information technology, electronics and photonics, and biotechnology. Manufacturing is a subset in all four categories. Technical topics under each of the main categories are broad and diverse.
The ATP selects projects through a competitive, peer-review process. Technical and business reviewers evaluate each proposal against ATP’s published selection criteria. Technical reviewers look for significant innovation in technology and high technical risk, as well as feasibility, quality of the R&D plan, and the experience and qualifications of the technical staff assigned to the project. Business reviewers look for the potential of the proposed technology to produce broad-based economic benefits to the nation, the need for ATP funding support, the proposed pathway to commercialize the technology and deliver economic benefit, and the experience and qualifications of the business staff assigned to
For the U.S. Small Business Administration’s definition of “small business,” see <http://www.sba.gov/services/contractingopportunities/sizestandardstopics/faqs/index.html>.
the project. Although ATP encourages companies to plan for commercialization of technology from the start of the project, ATP will not pay for product development or activities related to commercialization; these expenses are left to the private sector.
The ATP has been active in supporting entrepreneurial startup firms. The role of ATP as a public-private partnering program extends from providing critical funding to early-stage technology projects and includes encouraging collaboration among firms and other organizations, fostering information exchange, and facilitating technology entrepreneurship activities.
THE ADVANCED TECHNOLOGY PROGRAM—THEORY3
Rationale for Public Support of Early-Stage Technology Development
The rationale for public-private partnerships is that there exists a funding gap for entrepreneurs who seek transition from scientific invention to commercial innovation. Some argue that only minimal intervention by the government is needed to ensure economic efficiency. This argument assumes that there is perfect information and thus Adam Smith’s invisible hand leads to efficient outcomes. Joseph Stiglitz (2005) notes that in a market economy with imperfect and asymmetric information and incomplete markets, economies are not efficient on their own, which “leads to the conclusion that there is a potentially significant role for government.” Arrow (1962) argued that there exist numerous market failures, especially in the market for new ideas and technological information. Market failures exist when “we expect a free enterprise economy to under-invest in invention and research (as compared to the ideal) because it is risky, because the product can be appropriated only to a limited extent, and because of increasing returns in use.” Arrow highlights the role of uncertainty, which leads to contracting problems that result in a funding gap. The conversions of inventions to commercial innovations face many obstacles and risks.
The fundamental rationale for government support of R&D rests on the idea that the social rate of return on R&D investment is greater than the private rate of return. That is, the overall benefit to society, when all benefits are considered, exceeds the private benefit that accrues to the individual firm that performs the R&D. What this means from the policy perspective is that the private sector or individual firm does not have as much incentive to carry out R&D as is socially optimal because it cannot capture all of the benefits of its R&D investment (Mansfield 1996; Griliches 1993; Stiglitz 2005).
The overall benefit of R&D exceeds the private individual return because much of the benefit of R&D accrues to those other than the company carrying out the R&D. The innovating company captures a portion of the total value generated
by a new technology in the form of new profits. But an additional large portion of the total value of a new technology is not captured by the innovating company, but by other firms inside and outside of the industry of the innovating companies.
Downstream Value from New Technology
Downstream users and consumers receive benefits when they adopt new technology introduced by innovating companies. This value accrues to users and consumers, and not to the innovating companies. Since companies cannot capture all of the value from R&D that leads to new technology, they will not pursue projects that have substantial and broad benefits. These projects do not offer sufficient private profits for a company to justify its private investment. These are cases for public-private partnership. A public program such as the ATP can partner with private industry to provide the funding necessary to carry out the R&D and technology development that has high potential for broad economic benefit.
Knowledge Spillovers From R&D
An important aspect of investment in R&D is that the knowledge benefits tend to “spill over” to others not directly involved in the original R&D work. When one company conducts research, other companies also receive benefits because the results of R&D often become more generally known through patents, publications, and other means of industry knowledge dissemination. Researchers can learn from research conducted at other companies and become more productive in their own research endeavors. Because of such knowledge spillovers, when one company conducts R&D, the overall benefit is greater than what this one company receives. Because other companies (as well as consumers and the general public) also benefit, there is a strong policy rationale for encouraging company R&D investment through a public program such as ATP.
Empirical Estimates of the Social Rate of Return on R&D
A number of economic studies have assessed the magnitude of the social rate of return on R&D investment and the extent to which the social return exceeds the private rate of return. Although any given study may have limitations, the general conclusion from the combined body of evidence is that the social rate of return on R&D is much higher than the private rate of return4 (Griliches 1993; Jaffe 1998). In absolute terms, the social rate of return on R&D is most likely 15 to 30 percentage points greater than the private rate of return. In relative terms, the social rate of return is most likely 50 to 100 percent the magnitude of the private rate of
return. The empirical evidence from economic studies therefore shows that there is a strong case for public support of industrial R&D and that the social benefits from R&D are very great (Mansfield 1996).
The Funding Landscape for Early-Stage Technology Development
The ATP’s niche in the U.S. innovation system as a funding source for early-stage technology development lies between basic scientific research and commercial product development. Basic research is publicly supported by government because benefits from basic research are broadly diffused; the benefits are largely social benefits not limited to any person, firm, or organization conducting the research. Commercial product development, on the other hand, is carried out by companies motivated by the opportunity for private profits. Early-stage technology development is situated between these two activities and is characterized by high technical and business risk to the innovating company—since outcomes are uncertain and returns are far in the future—and also high potential for delivering great social value and private returns from the successful development of promising new technology. It is in this “space” that ATP plays an important role in supporting and funding new technology development.
How important is the public role in funding early-stage technology development? Who are the players in funding early-stage technology development? In a report commissioned by ATP’s Economic Assessment Office (EAO), Branscomb and Auerswald (2002) estimate that between $5 billion (2 percent) and $36 billion (14 percent) of overall national R&D spending in 1998 was devoted to early-stage technology development. The relatively small share of total national R&D spending devoted to early-stage technology development supports the view that there is a “funding gap” where the amount of funding currently available is less than what is socially optimal or desirable. As discussed earlier, there is good theoretical and empirical reason to support the policy view that a higher level of national R&D is well justified.
Further, Branscomb and Auerswald find that private equity “angel” investors, corporations, and the federal government are the main sources of funding for early-stage technology development (see Figure 2). Of particular interest is the finding that the federal government, and not organized venture capital, is a major funding source for early-stage technology development. Approximately 20 to 25 percent of early-stage technology development is funded by the federal government, with the ATP as one of the principal federal programs focused in this critical area.
The ATP partners with small firms and startup firms, as well as larger firms, in supporting early-stage technology development. As seen in Figure 2, the other main funders of early-stage technology development are angel investors who fund startup firms led by entrepreneurs, and industry corporations that fund ongoing R&D efforts in their core competencies. This shows the importance for ATP
partners with both small startup firms and more established firms, and also highlights how ATP can partner with angel investors and corporate funding sources in accelerating and supporting the development of early-stage technologies.
For small startup firms, the ATP award provides critical funding support as well as benefits that extend beyond funding. The ATP has been very active in supporting small firms and entrepreneurial startup firms. Almost two-thirds of ATP awardees are small companies with fewer than 500 employees. Of these small firms, about 25 percent are very small firms with fewer than 10 employees, and another approximately 40 percent have from 10 to 20 employees. An ATP award provides external validation of the firm’s technology, and increases the visibility of the company, both of which help the company attract additional funding from other sources. Survey data on ATP-awarded companies indicate that the ATP award does help attract additional funding to the company’s technology (Feldman and Kelley 2001; ATP 2003a and 2005b).
Approximately 30 to 47 percent of early-stage technology development is funded by corporations (see Figure 2). Among corporations, the fraction of R&D that is dedicated to early-stage technology development varies both among firms and within industries. One estimate is that overall corporate spending on early-stage technology development is $13.2 billion, or about 9 percent of
total corporate R&D. Of this $13.2 billion, early-stage technology investment in the computer software industry is $0.1 billion or 0.75 percent, whereas for the biopharmaceutical industry the average rate is 13 percent (ranging from 0 to 30 percent in the biopharmaceutical companies interviewed). A key driver appears to be the life-cycle position of the industry and individual company. More mature industries, such as the automotive sector, tend to invest a smaller percentage of R&D in earlier stages than do industries at an earlier stage of evolution, such as the biotech industry. Policies to encourage early-stage technology development may be most effective when directed to encouraging corporations to undertake higher risk research in new business areas outside their core. Monsanto’s move into genetics in the 1980s is an example of a company making a temporary movement backwards out of product development and into a strategy emphasizing basic and early-stage technology development research (Branscomb and Auerswald 2002; Auerswald et al. 2005).
THE ADVANCED TECHNOLOGY PROGRAM—EVIDENCE THAT THEORY AND PRACTICE WORK
Evaluation works best when it is closely mapped to a program’s mission. ATP’s legislative mandate is to increase the prosperity of the United States by funding the development of high-risk technologies through a public-private partnership. ATP’s goals are to add to the nation’s scientific and technical knowledge base, to foster accelerated technology development and commercialization, to promote collaborative R&D, to refine manufacturing processes, to ensure small business participation, to increase the competitiveness of U.S. firms, and to generate broad economic and social benefits (Feller and Ruegg 2003).
ATP award recipients deliver benefits directly and indirectly. Direct benefits are achieved when technology development and commercialization is accelerated, which leads to private returns and market spillovers. Indirect benefits are delivered through publications, conference presentations, patents, and other ways in which knowledge is disseminated. From program purpose and design to final outputs, outcomes, and impacts, ATP’s evaluation program measures these direct and indirect benefits.
The National Research Council has praised ATP’s evaluation program, stating that “The ATP assessment program has produced one of the most rigorous and intensive efforts of any U.S. technology program … the quality, quantity, and analytical range of [their] studies are impressive” (National Research Council 2001).
Evaluation has been an integral part of program operations from the outset. To learn about the program’s impact, program officials set aside a small amount of ATP’s initial budget in 1990 to fund rudimentary evaluation activities. Since then, the budget for program evaluation has grown significantly, as has interest in evaluation. With a professional staff of economists, statisticians, information specialists, social scientists, business liaison specialists, and administrative support,
the ATP’s EAO is charged with carrying out ATP’s evaluation activities. EAO aims to measure the economic impact of ATP’s funding of high-risk, enabling technologies and also to increase understanding of underlying relationships between technological change and economic phenomena. EAO also provides business and economic expertise for ATP selection boards and locates expert business reviewers to review proposals.
ATP’s evaluation program goals are to meet external requests for ATP program results, to use evaluation as a management tool to meet program goals and to improve program effectiveness, to understand ATP’s contribution to the U.S. innovation system, and to develop innovative methodologies to measure the impact of public R&D investment. EAO tracks progress throughout the life of funded projects and for several years after the ATP funding ends. Evaluation work consists of conducting surveys, compiling data, producing statistical analyses, undertaking economic and policy research studies, and commissioning studies by consultants and research economists.
ATP’s Evaluation Best Practices
ATP’s experience in funding early-stage technologies and evaluating the impact of its awarded projects has resulted in many best practices. These best practices may prove useful to similar government programs in the early stages of their operations or to government programs that must meet external performance reporting requirements (Chang, Shipp, and Wisniewksi 2005).
Committing to Performance Evaluation
One of the most important best practices is to establish the practice of evaluation and to sustain those activities despite budgetary pressures. ATP allocates funding for a staff dedicated to evaluation activities—the EAO—and for carrying out evaluation activities using internal and external resources. It is important for public research and development programs to treat evaluation as a core activity and to pursue evaluation within a framework that measures the program against its stated objectives. Having a dedicated staff with appropriate backgrounds, capabilities, and experience is essential; having a dedicated budget for evaluation activities is critical.
Using a Multifaceted Approach to Evaluation
ATP’s evaluation tools assess commercialization as well as knowledge creation and dissemination. These methods must accommodate the measuring of inputs, outputs, outcomes, and impacts over the life cycle of a project. Research and development takes place in the short to mid term, commercialization in the mid to longer term, and widespread diffusion of the technology over a longer time
horizon. This time frame varies by technology area—shorter for information technology projects and much longer for biotech projects (Powell and Moris 2002). It also accounts for why multiple evaluation approaches are needed to capture the status of projects at various stages of their life cycle.
Commissioning External Studies by Experts
ATP contracts with experts to conduct economic analysis of individual projects, clusters of projects, or concepts underlying the economic principles of the program. ATP’s EAO works with well-known researchers to shape, manage, and produce many of its reports. In the early years, EAO worked with economists affiliated with the National Bureau of Economic Research to help lay a strong foundation for evaluating the program. Zvi Griliches, Edwin Mansfield, Adam Jaffe, Bronwyn Hall, and others collaborated with ATP on important research to explore how to measure and track key economic concepts that apply to government support for the development of high-risk, enabling technologies carried out by the private sector. They studied concepts such as spillovers (knowledge, network, and market spillovers, see Jaffe 1997), return on investment (social, private, and public rates of return, see Mansfield 1996), and research productivity. By supplementing core in-house evaluation capability with expertise provided by outside contractors, ATP has pursued a balanced approach to evaluation and has welcomed new ideas and approaches.
Evaluating Unsuccessful Projects
Another best practice is evaluating unsuccessful projects along with successful ones. There is a great deal to learn from projects that failed to complete their goals or to deliver promised benefits. ATP has analyzed the reasons behind projects terminating early (ATP 2001, Appendix B). The knowledge generated by examining the reasons why projects fail can enhance project selection and project management.
Almost 10 percent of projects terminate early. A project can end early or not start for participant-initiated reasons, such as a change in goals, financial distress, lack of technical progress, or the inability of a joint venture project to reach an agreement on rights to intellectual property. A project can also end for ATP-initiated reasons, such as the project’s failure to meet ATP project selection criteria or its shift away from the pursuit of high-risk research. In a very few cases, early success was the cause for early termination.
Strategically Presenting Results
Results have more effect if they are presented so that a nontechnical person can understand the science and commercialization. Results are presented in mul-
tiple ways—a brief abstract enabling someone to quickly grasp the key findings, an executive summary for someone who wants an overview of key highlights, and the full report. Quantitative findings are presented in tabular form, with graphics, and with accompanying qualitative analyses. Many findings are released in fact sheets and made available on ATP’s Web site. ATP has also published three special topic brochures that highlight projects in the health care, energy, and manufacturing sectors (ATP 2003b, 2003c, 2005).
Another way that results and data are summarized is in the form of a statistical abstract, an idea borrowed from the U.S. Census Bureau’s annual Statistical Abstract. ATP’s biennial statistical abstract was first released in September 2004, in a report called Measuring ATP Impact, 2004 Report on Economic Progress. The report describes ATP, using findings and data from recent reports and statistics. It also provides summaries of recent studies and ten detailed statistical tables that provide data on number and distributions by types of awards, technology areas, geographic regions, university participation, number of patents, commercialization, and post-award attraction of external funding.
Developing Innovative Methods to Evaluate ATP’s Effectiveness
Evaluation of emerging technologies is a relatively new field. While traditional economic and social science methods can be employed to assess program success, the existing tools are often insufficient to describe the nuances and input of public-private investments. It is appropriate to modify existing tools, develop exciting new tools, or combine existing methods in ways never before explored.
For example, one of the more difficult concepts to measure is social return resulting from an ATP project. Social return includes private returns to the participating company in the project, and public returns, including knowledge, network, and market spillover benefits to that company’s customers or to other firms, and a variety of indirect benefits to other companies and their customers as a result of the diffusion of knowledge created from the project (see Jaffe 1997 and Chang, Shipp, and Wang 2002 for a historical description of this issue).
Despite the difficulties in measuring social return, ATP has pursued a greater understanding of this concept by collaborating with consultants, professional economists, and academicians. Together, they carry out retrospective and prospective benefit-cost studies of a range of technologies and projects to test and stretch various methodological approaches. These studies include case studies of projects that developed photonics technology for use in petroleum refining, building controls, emergency medicine, and industrial materials (Pelsoci 2005), flow-control machining technology (Ehlen 1999), and technologies that reduced the dimensional variation of U.S. motor vehicles (Polenske et al. 2004).
In measuring spillovers, for example, they have used various approaches and means of illustration. To capture knowledge spillovers for the status reports
of completed projects—a portfolio-wide, mini case study tool—they developed patent trees that illustrate multi-tiered citations of patents that were issued for ATP-funded technologies. In addition, a study was commissioned to examine knowledge spillovers using social network analysis. This emerging method uses fuzzy logic and systems analysis to examine knowledge spillovers from research and development projects within networks of participating organizations (see the discussion in Ruegg and Feller 2003, pp. 271-275 and Fogarty et al. 2006).
To study market spillovers, they explored the use of the U.S. Department of Commerce’s Bureau of Economic Analysis input-output tables. Specifically, the first 50 completed ATP projects were mapped to their make-and-use industries to trace where the new technologies began and where they have since ended up (Popkin 2003). They are also exploring other emerging methods to measure spillovers and the impact of ATP funding, including coding potential commercial applications identified by ATP project participants using NAICs (North American Industry Classification) codes to identify make-and-use industries that illuminate the spillover path (Nail and Brown, forthcoming).
Systematically Collecting Data
The cornerstone of ATP’s evaluation program is its comprehensive survey and data collection system. Survey collection efforts are structured to align with overall evaluation goals, which in turn are crafted to optimize the performance of ATP. As part of an ongoing survey and database assessment effort, we have identified six broad-based goals that form the conceptual basis of our surveys: (1) opportunities for national economic benefits, (2) acceleration of R&D, (3) increased investment in high-risk, long-term technology, (4) stimulation of collaboration, (5) progress in commercialization of technology, and (6) longer-run changes in firm behavior that result from participating in an ATP project. These goals define how ATP projects affect the economy and society.
ATP surveys can be viewed as a microcosm of our overall evaluation program at ATP. The survey system is a multifaceted effort that is designed to meet multiple (but complementary) program goals while balancing efficiency in operation with excellence in results. This is achieved by identifying and leveraging both internal and external resources and harnessing the benefits of collaboration with survey experts (a tactic learned through our evaluation of ATP), and by relying on continual self-assessment and feedback. We do not lose sight of our goal to measure against mission, in the short, medium, and long term.
Baseline information is collected on the initial survey, and follow-up questions in each area are included at the appropriate anniversary, closeout, or post-project surveys. The surveys collect data on diffusion of knowledge (patents, publications, presentations, and other information about intellectual property); measures of social and environmental effects (spillovers); acceleration in terms of reduced time to achieve technical progress and time-to-market; collaboration
arrangements with universities, other firms, and other organizations; national economic benefits (business growth, development of business relationships and networks); the diversity of commercial applications arising from the technology that ATP has funded; and commercialization progress and expectations of revenues from commercialization of the technology, licensing, and cost-savings.
Finally, ATP surveys capture commercialization progress, results, and expectations, or specific aspects of it, although ATP does not fund the commercialization phase of projects. Our mission at ATP is not simply to fund high-risk technologies, but to fund high-risk technologies that have a strong potential to enhance economic growth. Economic growth can only be achieved when the technology enters the marketplace. To measure this impact, information is collected from firms on current and expected economic value achieved through revenues from commercial applications of the technology, licensing, and cost savings.
Examples of ATP’s Studies
As noted above, ATP uses a multifaceted approach to evaluation. To reflect the richness and diversity of the studies, several examples, including highlights of their findings, are presented below. These examples include policy studies, status reports of completed ATP-funded projects, selected survey results, and benefit-cost studies. These examples are provided to show the depth and breadth of our assessment work.
Project and Portfolio Assessment
Status reports are descriptive mini case studies for each completed ATP project written several years after ATP funding ends. Status reports address how well the project performed against ATP’s mission objectives to create and disseminate knowledge via acceleration, collaboration, commercialization, and benefits to the economy beyond the firm or firms developing the ATP technologies.
A performance rating for each project (zero to four stars) is computed using a uniform set of data.5 The aggregation of stars provides a portfolio view of ATP performance. For example, aggregating the performance ratings for the first 150 ATP projects shows the distribution shown in Table 1.
The largest group of projects, 28 percent, fell into the two- and three-star categories. The three-star projects show strong progress. Combining these with the 13 percent rated outstanding shows an impressive 41 percent of projects performing at a high level. The two-star projects show moderate progress but are not particularly robust overall. Thirty percent of the projects scored one star or less, which is not surprising, given that ATP projects are high-risk R&D and not all
For examples of ATP studies and a description of the methodology used to determine the performance rating, access the NIST Web site at <http://www.atp.nist.gov/eao/ir05-7174/chapt5.htm>.
TABLE 1 Distribution of Performance Ratings for First 150 ATP Projects
Percent of Projects
projects are expected to succeed. Projects may fail for technical reasons, business reasons, or a combination of both.
Each status report also includes a patent tree for each patent filed during ATP project or after the project ends to show the citation of the patent in subsequent patents. Patent trees are updated annually to demonstrate that knowledge spillovers continue several years after the ATP project ends.
A study of the first 150 completed ATP projects shows that 203 new products or processes resulted from 91 of these projects, and employment changes for three out of five of the small companies were quite large. Forty-nine companies at least doubled in size, and 22 companies grew by more than 500 percent.
Answering the Counterfactual Question: What Happens Without ATP?
The Survey of ATP Applicants was conducted for awardees in the 1998, 2000, and 2002 competitions and is planned again for applicants from the 2004 competition. This survey was administered to all applicants in the previous competition year to compare the company and project characteristics of awardee and nonawardee companies soon after awards are announced. It addresses the counterfactual question: What happens when a project does not receive ATP funding? The survey results found that 39 percent of those projects were not pursued, and 44 were pursued on a smaller scale. Of those pursued on a smaller scale, more than four out of five reported that their project scope was reduced to below 40 percent of the proposed ATP project.
Evidence from the Survey of ATP Applicants shows that ATP is successful in directing funding to projects that have higher technical risk and longer time horizons than projects proposed by nonawardees (ATP 2003a). A measure of technical risk is the probability that a project will not achieve its technical goals.
Among ATP awardees, the average estimate for the probability of not fully achieving technical goals is 45 percent, compared to nonawardees’ estimated probability of 31 percent.
More than half (54 percent) of ATP awardees expect a time horizon of four years or more on their proposed ATP projects, compared to one-third of nonawardees.
Proposed ATP projects for both awardees and nonawardees are higher risk and have a longer time horizon than do typical R&D projects. ATP awardees report a greater contrast between their proposed and typical R&D projects, compared to nonawardees.
A key finding is that ATP awardees attract additional funding after submitting their ATP proposal. This phenomenon is referred to as the halo effect. For example, three out of four awardees report increased internal funding, whereas one out of four nonawardees reports increased funding from internal company sources. ATP awardees are also more likely to receive funding from external sources. One out of three awardees reports increased funding, and only one out of five nonawardees report increased funding from external sources (Feldman and Kelley 2001; ATP 2003a and 2005b).
Measuring Acceleration Effects
The Business Reporting System (BRS) allows an examination of ATP awardees from a longitudinal perspective and from a cross-sectional perspective. Responses to the BRS surveys indicate that ATP funding accelerated R&D in nine out of ten organizations. Of those organizations that indicated that they were ahead in their R&D cycle:
Thirteen percent indicate they are ahead by one year.
Fifty-three percent indicate that they are ahead by one to three years.
Seven percent indicate that they are ahead by more than three years.
ATP participants report that the acceleration of R&D reduces the time it will take to bring products to market or to implement new production processes. Reduction in time-to-market by two years or more is anticipated for about three out of five planned commercial applications.6
Measuring Outcomes—Benefit-Cost Studies
Benefit-cost studies are one of the primary ways to measure outcomes quantitatively. Outcomes are difficult to measure because one must make assumptions about the impact of the new technology and acceptance by buyers. These studies examine one project or a cluster of projects in the same technology area to assess both retrospective (realized benefits) as well as future benefits. A qualitative analysis is also included in the studies.
One example of a benefit-cost study is Low-Cost Manufacturing Technology for Amorphous Silicon Detectors, a joint venture project funded by ATP in 1995
(Pelsoci 2003). Digital mammography and radiography systems are innovative technology solutions to the diagnostic and productivity limitations of conventional x-ray systems. The new process, which was implemented in 2004, is expected to reduce fabrication costs by approximately 25 percent without compromising performance. On the basis of 33 million mammography and 68 million chest x-rays per year, prospective benefits are $125 to $193 for every $1 that ATP has spent. Societal benefits include avoidance of unnecessary medical procedures as a result of lower false-positive rates, improved breast cancer detection, reduced patient exposure to radiation, and reduced examination time.
Behavioral additionality is defined as the difference in firm behavior that persists over time and results from a government intervention.7 The assumption is that government financing of business R&D changes firm behavior in a desirable direction. Behavioral additionality has generally been ignored by econometric studies of the effects of R&D support that focus on input additionality, where estimates are made of additional R&D expenditure, or output additionality, whereby firm performance is compared between recipients and nonrecipients of public support.
A one-time survey of joint ventures incorporated questions that asked about changes in firm behavior that resulted from having an ATP award. Many of the new questions are now incorporated into the ongoing BRS surveys.
Results from the Joint Venture Survey show that the formation of ATP joint venture projects and the rigor of the agreement fostered trust and cooperation among partners. Respondents reported that goodwill and trust were high among joint venture partners. A regression analysis showed ATP involvement to be an explanatory factor, along with effective governance procedures and the size of the joint venture (that is, the number of partners). Findings from another ATP survey, the Post-Project Survey, show persistent collaborative links, with 46 percent continuing to work with their partners on non-ATP technology and 14 percent with their subcontractors. More than half (55 percent) continued in R&D because of their positive ATP experience (Shipp et al. 2005). Participating in the Organisation for Economic Co-operation and Development (OECD) workshops on behavior additionality has provided ATP with new insights and approaches to our work and is especially timely as we are in the midst of improving our surveys. Including new questions that relate to behavior additionality will enhance and inform our future work.
Analysis of Regional Innovation Patterns
ATP has three projects under way to investigate regional innovation patterns. The first examines patent hot spots to identify geographic areas where there is intensive R&D in a specific technology area. Hot-Spot Analysis provides a filter on recent patents by focusing on the 20 percent of recent patents that are likely to have impact in the future. Using recent patents with no filtering mechanism is problematic because there were more than 300,000 patents issued in the past two years, and most of them may have little value. The project’s ultimate goal is to identify those patents that are more closely associated with high-risk, early-stage technology. Use of this method is mentioned in the legislation for the National Innovation Act of 2005 to identify areas for regional economic development.
A second project is examining regional patterns graphically—for example, identifying and plotting fiber optic installation as one measure of high-tech activity. The third project is the development of a database that collects data on economic activity by area and compares it to the U.S. average or other specified areas. This is a new area of research under way, and preliminary results are intriguing.
THE ADVANCED TECHNOLOGY PROGRAM—CONCLUSIONS
The ATP’s mission is to support U.S. companies in pursuing early-stage, high-risk research to develop new technologies that have great potential for producing broad-based national economic benefit. ATP funds industry-led research and development projects that have high technical risk and inventiveness and promising potential for broad economic impact. Since 1990, the ATP has been a significant government player in supporting the development of emerging technologies in the United States. The rationale for programs such as the ATP rests on theory and evidence that the social benefits of R&D are greater than the private returns. A hallmark of the ATP is its multifaceted and integrated evaluation program that uses economic and statistical analysis to develop estimates of impacts of ATP funding on project timing and success. Through the use of systematic evaluation, our surveys, studies, and reports show that ATP is indeed meeting its mission to accelerate the development of high-risk, enabling technologies.
Advanced Technology Program. 2001. Performance of 50 Completed ATP Projects, Status Report Number 2. NIST Special Publication 950-2. Washington, D.C.: U.S. Government Printing Office.
Advanced Technology Program and National Institute of Standards and Technology. 2003a. Survey of ATP Applicants 2000. NIST GCR 03-847. Gaithersburg, MD: National Institute of Standards and Technology.
Advanced Technology Program and National Institute of Standards and Technology. 2003b. Powering Our High-Speed Economy: A Profile of ATP Energy Investments. Gaithersburg, MD: National Institute of Standards and Technology.
Advanced Technology Program and National Institute of Standards and Technology. 2003c. Beyond Measure: A Profile of ATP Health Care Investments. Gaithersburg, MD: National Institute of Standards and Technology.
Advanced Technology Program and National Institute of Standards and Technology. 2004. Measuring ATP Impact: 2004 Report on Economic Progress. Gaithersburg, MD: National Institute of Standards and Technology.
Advanced Technology Program and National Institute of Standards and Technology. 2005a. A Profile of ATP Manufacturing Investments, Inspiring Innovations in Industry. Gaithersburg, MD: National Institute of Standards and Technology.
Advanced Technology Program and National Institute of Standards and Technology. 2005b. Survey of ATP Applicants 2002. NIST GCR 05-876. Gaithersburg, MD: National Institute of Standards and Technology.
Anderson, Gary, Jeanne Powell, and Stephanie Shipp. 2003. “Improving the Advanced Technology Program’s Business Reporting System—A Firm-Level R&D Survey.” Proceedings of the American Statistical Association, Government Statistics Section, [CD-ROM]. Alexandria, VA: American Statistical Association.
Arrow, Kenneth. 1962. The Rate and Direction of Inventive Activity. Princeton, NJ: Princeton University Press.
Auerswald, Philip E., Lewis M. Branscomb, Nicholas Demos, and Brian K. Min. 2005. Understanding Private-Sector Decision Making for Early-Stage Technology Development: A “Between Invention and Innovation Project” Report. NIST GCR 02–841A. Gaithersburg, MD: National Institute of Standards and Technology.
Branscomb, Lewis, and P. Auerswald. 2002. Between Invention and Innovation: An Analysis of Early-Stage Technology Development. NIST GCR 02-841. Gaithersburg, MD: National Institute of Standards and Technology.
Chang, Connie, Stephanie Shipp, and Andrew Wang. 2002. The Advanced Technology Program: A Public-Private Partnership for Early-stage Technology Development. Venture Capital 4(4): 363-370.
Chang, Connie, Stephanie Shipp, and Lorel Wisniewski. 2005. Evaluation Best Practices and Results: The Advanced Technology Program. NIST 05-7541. Gaithersburg, MD: National Institute of Standards and Technology.
Darby, Michael, Lynne G. Zucker, and Andrew J. Wang. 2002. Program Design and Firm Success in the Advanced Technology Program: Project Structure and Innovation Outcomes. NISTIR 6943. Gaithersburg, MD: National Institute of Standards and Technology.
Ehlen, Mark A. 1999. Economic Impacts of Flow-Control Machining Technologies: Early Applications in the Automobile Industry. NISTIR 6373. Gaithersburg, MD: National Institute of Standards and Technology.
Feldman, Maryann P., and Maryellen R. Kelley. 2001. Winning an Award from the Advanced Technology Program. NISTIR-6577. Gaithersburg, MD: National Institute of Standards and Technology.
Fogarty, Michael S., Amit K. Sinha, and Adam B. Jaffe. Forthcoming. ATP and the U.S. Innovation System—A Methodology for Identifying Enabling R&D Spillover Networks with Applications to Micro-electromechanical Systems (MEMS) and Optical Recording. Draft report submitted to ATP.
Griliches, Z. 1993. “The Search for R&D Spillovers.” Scandinavian Journal of Economics 94(S): S29-S47.
Jaffe, Adam B. 1997. Economic Analysis of Research Spillovers: Implications for the Advanced Technology Program. NIST GCR 97-708. Gaithersburg, MD: National Institute of Standards and Technology.
Mansfield, Edwin. 1996. Estimating Social and Private Returns from Innovations Based on the Advanced Technology Program: Problems and Opportunities. GCR 99-780. Gaithersburg, MD: National Institute of Standards and Technology.
Nadiri I. 1993. Innovations and Technological Spillovers. NBER Working Paper No. 4423. Boston, MA: National Bureau of Economic Research.
Nail, John, and Hayden Brown. Forthcoming. Classifying and Analyzing ATP Projects Using the North American Industry Classification System (NAICS). NISTIR 2005. Gaithersburg, MD: National Institute of Standards and Technology.
National Research Council. 1999. The Advanced Technology Program: Challenges and Opportunities. Charles W. Wessner, ed. Washington, D.C.: National Academy Press.
National Research Council. 2001. The Advanced Technology Program: Assessing Outcomes. Charles W. Wessner, ed. Washington, D.C.: National Academy Press.
Pelsoci, Thomas. 2003. Low-Cost Manufacturing Process Technology for Amorphous Silicon Detector Panels: Applications in Digital Mammography and Radiography. NIST GCR 03-844. Gaithersburg, MD: National Institute of Standards and Technology.
Pelsoci, Thomas. 2005. Photonic Technologies: Applications in Petroleum Refining, Building Controls, Emergency Medicine, and Industrial Materials Analysis. NIST GCR 05-879. Gaithersburg, MD: National Institute of Standards and Technology.
Polenske, Karen, Nicolas Rockler, and others. 2004. Closing the Competitive Gap: A Retrospective Analysis of the ATP 2mm Project. NIST GCR 03-856. Gaithersburg, MD: National Institute of Standards and Technology.
Popkin, Joel. 2003. Inter-Industry Diffusion of Technology That Results from ATP Projects: Preliminary Research of the Potential Impacts of ATP Funding. NIST GCR 03-848. Gaithersburg, MD: National Institute of Standards and Technology.
Powell, Jeanne, and Francisco Moris. 2002. Different Timelines for Different Technologies. NISTIR 6917. Gaithersburg, MD: National Institute of Standards and Technology.
Ruegg, Rosalie, and Irwin Feller. 2003. A Toolkit for Evaluating Public R&D Investment: Models, Methods, and Findings from ATP’s First Decade. NIST GCR 03-857. Gaithersburg, MD: National Institute of Standards and Technology.
Sakakibara, Mariko, and Lee Branstetter. 2002. Measuring the Impact of ATP-Funded Research Consortia on Research Productivity of Participating Firms, A Framework Using Both U.S. and Japanese Data. NIST GCR 02-830. Gaithersburg, MD: National Institute of Standards and Technology.
Shipp, Stephanie, Andrew Wang, Steve Campbell, Lorel Wisniewski, Kerry Levin, Jennifer O’Brien, and Westat. 2005. “Measuring Behavior Additionality in ATP Joint Venture Projects: Findings from the U.S. Advanced Technology Program.” Paper presented at OECD Workshop on Behavioral Additionality, Vienna, Austria, January 31-February 1, 2005.
Solomon Associates. 1993. The Advanced Technology Program: An Assessment of Short-Term Impacts: First Competition Participants. Gaithersburg, MD: National Institute of Standards and Technology.
Stiglitz, Joseph. 2005. “The Ethical Economist.” Foreign Affairs Council on Foreign Affairs. November/December. Accessed at <http:www.foreignaffairs.org>.
U.S. Congress. 1988. Omnibus Trade and Competitiveness Act of 1988 (P.L. 100-418, codified in 15 U.S.C. 278n.) and later amended by the American Technology Preeminence Act of 1991 (P.L. 102-245, codifies in 15 U.S.C. 3701).