Page 173

V

RESEARCH PAPERS



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 173
Page 173 V RESEARCH PAPERS

OCR for page 173
Page 174

OCR for page 173
Page 175 The ATP Competition Structure Alan P. Balutis and Barbara Lambis 1 National Institute of Standards and Technology INTRODUCTION: THE ADVANCED TECHNOLOGY PROGRAM The Advanced Technology Program (ATP) of the Department of Commerce's National Institute of Standards and Technology (NIST) provides cost shared funding to industry to accelerate the development and broad dissemination of challenging, high-risk technologies that promise broad-based economic benefits for the nation. 2 The ATP allows industry to extend its technological reach by funding: Emerging and enabling technologies that make possible the development of future new and substantially improved products, industrial processes, and services in diverse areas of application; Technologies for which challenging technical issues stand in the way of success; Technologies whose development often involves complex “systems” problems requiring a collaborative effort by multiple organizations; 1 Alan P. Balutis is Director of the Advanced Technology Program. Barbara Lambis is the Senior Policy and Operations Advisor to the Director of ATP. This article was based on a draft prepared by Barbara Newland, who served as the ATP Competitions Manager until leaving government service on 11 November 2000. 2 The ATP was established in 1990 under the Omnibus Trade and Competitiveness Act of 1988 (Pub. L. 100-418), as amended by the American Technology Preeminence Act of 1991 (Pub. L. 102-245).

OCR for page 173
Page 176 Technologies that, because they are risky, are unlikely to be developed or proceed too slowly to compete in rapidly changing world markets without the ATP. The ATP funds technical research, but not product development (which is the responsibility of the private sector). The ATP is designed to be driven by industry: for-profit companies conceive, propose, co-fund, and execute all of the projects of which the ATP shares the cost. For-profit U.S. companies may apply to the ATP either as single companies or as joint ventures. Single company projects, though proposed by a single company, usually involve other companies, universities, or other organizations as subcontractors or informal collaborators. Joint-venture projects must have at least two for-profit company participants. Joint ventures may include other members, including other companies of any size, governmental laboratories, universities, and other nonprofit organizations. Joint-venture members may also have subcontractor and informal relationships with others. Sharing of costs is the essence of the program, and the cost-sharing requirements differ depending on how a project is structured. Single company recipients can receive up to $2 million for R&D activities for up to 3 years. ATP funds may only be used to pay for direct costs for single company recipients. Single companies are responsible for funding all of their overhead/indirect costs. Large, “Fortune 500” companies participating as single company recipients must cost share at least 60 percent of the total project costs. Joint ventures can receive funds for R&D activities for up to 5 years with no funding limitation other than the announced availability of funds. Joint ventures must cost share more than 50 percent of the total project costs. More than half of ATP awards have gone to single small company recipients or joint ventures led by small companies. On the average, ATP projects last a little more than three years. Figure 1 illustrates the two ways companies may apply. A U.S.-incorporated company (subsidiary) of a foreign-owned parent company which is incorporated in another country may apply if the company meets the conditions in the ATP legislation. 3 Prior to a final award, a foreign eligibility finding is made by NIST which involves the collection of evidence that: (1) the company's participation in the ATP is in the economic interest of the United States; (2) the country of incorporation of the participant's parent company affords U.S.-owned companies opportunities comparable to those afforded to any other company to participate in government-funded programs similar to ATP; 3 See 15 U.S.C. Sec. 278n. (d)(9) and regulations (15 CFR 295.3).

OCR for page 173
Page 177 ~ enlarge ~ FIGURE 1 The ATP application process (3) the country of incorporation of the participant's parent country affords U.S.-owned companies local investment opportunities comparable to those afforded to any other company; and (4) the country of incorporation of the participant's parent country affords adequate and effective protection for the intellectual property rights of U.S.-owned companies. ATP projects are selected through a highly competitive process. All proposals are selected based on a peer-review process using the established, transparent selection criteria. Only those proposals that are found to have both high technical and economic merit are selected for funding. 4 The quality of the selection process is critical to the program's success in several ways. A sense of fairness and rationality in the selection process is important to the willingness of proposers to participate. The ability to select well and with efficiency is critical to the program's performance and its ability to achieve its goal of substantial net benefits to taxpayers. This paper reviews the competitive structure of ATP's project selection process. It describes the basic principles that determine how the selection process is structured, and identifies the core components of all of the program's competi- 4 For more information about ATP, its selection process, and selection criteria, see the NIST ATP web site: http://www.atp.nist.gov .

OCR for page 173
Page 178tions to date. It also reviews the evolution of ATP's competitive structure since the first competition in 1990. KEY PRINCIPLES AND COMPONENTS IN PROJECT SELECTION Key Principles In establishing the process of project selection, ATP is guided by several underlying principles: Responsiveness. The competition structure must be responsive to industry, since U.S. businesses are the agents through which ATP achieves its mission. Integrity. The process must have integrity such that all proposals get a fair and consistent peer-review and are judged solely on how their proposal meets the published selection criteria. Scalable. The process must be scalable to allow for changes in the volume of proposals received, and revised in a timely manner to support growth. Evaluation. The process must produce the desired outcome; that is, a portfolio of projects that support the program's goals. Key Components In carrying out the project-selection process, the ATP uses several operational components: Announcements of ATP competitions and selection criteria alert industry of the opportunity and the challenge. Proposers' Conferences provide tutorials on the selection criteria and process. A Source Evaluation Board (SEB), a board made up of Federal employees, is convened to evaluate proposals against the selection criteria. Expert technical and business reviewers outside the SEB conduct independent peer reviews of proposals, which are provided to the SEB as advice in project-selection deliberations. All reviewers must be free from conflict of interest and must sign nondisclosure agreements. Outside technical reviewers are drawn from federal laboratories, and outside business/ economics reviewers are hired consultants, with suitably diverse expertise in business operations, venture capital, economics, and business development. Technology classification codes are used to identify the principal area of technology. This information is used along with non-proprietary abstracts of the proposal in finding suitable reviewers and may be used to form

OCR for page 173
Page 179 technical panels of SEBs. It is also used to track historical funding in various technology areas. Preliminary screening of all proposals identifies those with serious deficiencies, and these are eliminated from the competition. SEB deliberations, including evaluating proposals, as well as technical and business review; selection of semi-finalists, conducting face-to-face oral reviews with proposers, and ranking of proposals. Selecting Official makes final award selections. Debriefings provide feedback to unsuccessful proposers from SEB representatives on proposal strengths and weaknesses. THE PROJECT SELECTION PROCESS In brief, the process used over the past decade starts with ATP announcing a competition. One or more Source Evaluation Boards (SEB) are established to determine how proposals score against established selection criteria. A preliminary screening is performed for all proposals to identify those that have serious deficiencies and those that warrant further consideration. Independent reviews provided by outside technical and business experts are taken into account by the SEB in its consideration of each proposal. The SEB members discuss each proposal with reference to the selection criteria. If the Board members agree that a proposal is sufficiently strong in technical and business/economic merit, it is chosen to progress to the semi-finalist stage. Proposers selected as semi-finalist proposals may be invited to NIST for an oral review (a face-to-face discussion with the SEB). At this meeting, which typically lasts about two hours, the Board members question the proposers about the proposal. Questions involving technical, economic/business and budgetary aspects of the proposal, and, where applicable, questions about human and animal subjects in research are raised. There may be questions that have been raised by various reviewers at any stage of the proposal review process as well as questions that arise during the oral review. From this process of review of both written and orally presented information about proposed projects, semi-finalist proposals are ranked and the Selecting Official chooses funding recipients based upon the ranking, the availability of funds, adherence to the ATP selection criteria, and an appropriate distribution of funds among technologies and their applications. Award recipients are announced and awards are made in the form of cooperative agreements between the award recipients and NIST. Unsuccessful proposers are given the opportunity of telephone debriefings conducted with SEB members. The debriefing lets the proposers discuss their proposal's stronger and weaker points with those particularly knowledgeable, though the views represented reflect those of the entire Source Evaluation Board and provide insight into its deliberations. This information can be helpful if the company decides to revise and resubmit the proposal, submit another proposal to the ATP, or pursue the proposed research in some other way.

OCR for page 173
Page 180 The debriefing feature of the ATP is popular. A recent survey of 1998 proposers showed that 63 percent indicated that they had participated in a debriefing, and 69 percent of non-recipients said they found the debriefings to be “very helpful” or “reasonably helpful.” 5 EVOLUTION OF THE COMPETITIVE STRUCTURE The evolution of the competitive structure over the past decade is best understood when viewed against a backdrop of ATP's budgets, proposals received, and awards made, as illustrated in Figure 2, Figure 3, and Figure 4, respectively. The changing budgetary environment for the program has had a major impact. As Figure 2 indicates, through the initial years of the Bush administration, the program increased steadily, with substantial bipartisan support. In FY94-95, the program received substantial increases in funding. These were followed by a sharp decrease in FY96 from $340 million (after a recission) to $221 million. The program stayed in the $200 million range until FY2000, although the relative stability of this funding understates the political uncertainty associated with the program. These changes in budgetary levels have substantial influence on both the number of proposals submitted to the ATP and how the ATP organizes its competitive structure and operational processes. The evolution of the competition structure is described below. Four distinct periods are addressed: 1990-93, 1994-98, 1999-2000, and 2001. ~ enlarge ~ FIGURE 2 ATP appropriations, 1990-2000 6 5 The results of the survey are reported by M. P. Feldman and M. R. Kelley, “ Leveraging Research and Development: The Impact of the Advanced Technology Program,” in this volume. 6 The initial appropriation for 1995 was over $430 million which, after a recission by the new Congress, was reduced to $340.5 million in April 1995.

OCR for page 173
Page 181 General Competitions Only—1990-1993 From 1990 through 1993, the ATP was in its formative stage, with small but steadily rising budgets. Authorized for funding in 1988 during the Reagan Administration, the ATP received its first funding of just over $10 million in 1990 during the Bush Administration, and by 1993 its budget was more than $60 million. During these first four years, companies submitted in the range of 150-300 proposals per year, and the ATP made awards in the range of 10-30 per year. In each of these years, the ATP announced a single “General Competition” and convened an SEB composed of about a dozen members, some with technical backgrounds and some with business backgrounds. Given the size of the competition, a single SEB of this size was able to process all of the proposals. To provide advice to the SEB, the ATP obtained independent reviews from outside technical and business/economic expert reviewers. During these early years, the independent technical and business/economic reviews were conducted sequentially, with all of the proposals receiving technical reviews and only the semi-finalists receiving business/economic reviews. The independent technical reviewers were scientists and engineers drawn from government research laboratories, such as the National Institute of Standards and Technology, National Institutes of Health, National Science Foundation, Department of Energy Laboratories, and Department of Defense Laboratories. The first business/economic reviewers were primarily retired business executives. General and Focused Competitions—1994-1998 The Clinton Administration designated the ATP as an important element in its economic plan, and slated it for rapid growth. Dr. Arati Prabhakar, appointed Director of NIST, added Focused Program Competitions to the General Competition that the ATP had used previously. The idea was to add depth in selected technology areas, while maintaining the open door to great ideas from all areas of technology. Focusing in selected technology areas would allow greater advanced planning and a coordinated approach by industry and government. This approach was expected to assist the scale-up of the program and increase impact by accelerating the development of the selected technology areas overall. Opportunities Identified by Industry The ATP procedures presume that industry is closest to the market and to the potential of a technology. To draw on this expertise, the specific technology areas

OCR for page 173
Page 182for the focused competitions were chosen in a process that began with white papers submitted by industry. ATP staff then conducted public workshops to debate the suggestions and work with industry to refine the proposed technical scope. The merits of proposed focused competitions were judged on the following elements: Technical goals and program scope; Economic goals and scope of proposed work; Potential for broad-based economic benefits to the United States; Level of industry commitment and clear need for ATP support. Focused programs were selected that best met the ATP criteria and could be launched with the available funds. During the period 1994-98, it ran focused program competitions in the following areas: Adaptive Learning Systems; Advanced Vapor Compression Refrigeration Systems; Catalysis and Biocatalysis Technologies; Component-Based Software; Digital Data Storage; Digital Video in Information Networks; Information Infrastructure for Health Care; Manufacturing Composite Structures; Materials Processing for Heavy Manufacturing; Microelectronics Manufacturing Infrastructure; Motor Vehicle Manufacturing Technology; Photonics Manufacturing; Premium Power; Selective Membrane Platforms; Technologies for the Integration of Manufacturing Applications; Tissue Engineering; Tools for DNA Diagnostics. Several competitions were run for most of the focused-program areas, depending on industry interest, availability of funds, and continued robustness of the original goals. In 1994 there were six competitions (one general and five focused), in 1995 twelve competitions (one general and eleven focused), in 1996 one competition (one general), in 1997 seven competitions (one general and six focused), and in 1998 nine competitions (one general and eight focused). In 1994 and 1995 specific dollar amounts were allocated to each of the focused programs. (For example, in 1994 the Information Infrastructure for Health Care Focused Program was announced, with $185 million estimated as the cost over the several-year life of the program.) In total, the ATP awarded more than 150 cooperative agreements in focused program competitions.

OCR for page 173
Page 183 During this same period, the ATP also held a General Competition each year to ensure that great ideas from technology areas not addressed by the focused areas had an opportunity to be funded by the ATP. While it awarded the bulk of its funding through the Focused Competitions, it also had robust General Competitions each year on the order of those held when the program was smaller. The selection process for the General and Focused Competitions followed the same basic format as discussed above. Each competition was processed by its individual SEB, and there were as many SEBs as there were competitions. Open Competitions—1999-2000 In fiscal years 1999 and 2000, the ATP suspended its Focused Competitions and developed a hybrid model. This model combined the strengths of the focused program competitions, which industry preferred, with the General Competition. The resulting competition structure was dubbed “Open Competition” to highlight its difference from the tradition General Competition. A single SEB was used for the selection process; however, the SEB was divided into multiple technology boards, such that proposals in each technology area were batched and reviewed by a board focusing only on that technology area. The 1999 Open Competition, for example, used five technology boards (biotechnology, chemistry and materials, electronics and photonics, information technology, and manufacturing). Because most manufacturing technologies were found to cut across other technologies, the 2000 Open Competition merged the manufacturing proposals into the boards with the appropriate application area. The 1999-2000 change reflected several developments. The originally expected increase in the program funding to approximately the $1 billion level had failed to materialize. Consequently, there was insufficient funding to support industry's demands for multiple focused program topics. By this time there were more quality focused program recommendations than ATP could support. At the same time, some members of Congress were voicing stringent objections to the focused program mode of competitions, and annual uncertainties about funding levels continued. Competition-Dependent Variations in the Selection Process Although the basic selection process has remained relatively constant over the last decade, the type of competition has caused some variation in certain steps of the process. Examples follow: Preliminary Screening In Focused-Program competitions, each proposal was initially screened by the SEB to determine whether its technical scope was consistent with the appro

OCR for page 173
Page 184 ~ enlarge ~ FIGURE 3 ATP proposal submissions, 1990-2000 priate focused competition announcement. A proposal that was screened out at this point was sent to the General Competition's SEB for evaluation. Those that were screened out in any of the Competitions—General, Focused, or Open— due to other serious deficiencies, such as lack of a technical plan, were simply eliminated from competition. Generally about 10 percent have been rejected at this stage, for a wide variety of reasons. ~ enlarge ~ FIGURE 4 ATP awards, 1990-1999

OCR for page 173
Page 185 Technical and Business Reviews and Selection Proposals accepted into a Focused-Program competition went through the same reviews for business and technical merit as entries in General and Open Competitions. However, proposals in Focused-Program Competitions were reviewed in the context of a larger set of pre-planned technical and business goals and with a closer eye to synergy among projects within that program area. Attention to synergy was more difficult in General Competitions which one moment would be considering a biotechnology proposal and the next, a manufacturing technology. Open Competitions have no over-reaching larger program goals, but consideration of a group of proposals in the same technology area makes it more conducive to consider possible synergistic effects. Outside Independent Reviews During the first several years of General Competitions, the independent technical and business/economic reviews performed by outside reviewers to provide advice to the SEBs were performed sequentially. All proposals received technical reviews, but only semi-finalists received business/economic reviews by the reviewers outside the SEB. This process was changed to provide technical and business reviews in parallel, regardless of the type of competition, for two reasons: (1) to enable the debriefing process to cover business/economic strengths and weaknesses for all proposals rather than only semi-finalists, and (2) to provide adequate time for business/economic reviews to be conducted. CHANGES IN THE SUBMISSION PROCESS: OPEN COMPETITION AND ROLLING SUBMISSIONS In fiscal year 2001, ATP will again hold a major Open Competition. This year will see major changes in the proposal submission processes. The new Director of the ATP, Alan Balutis, introduced modifications to the ATP competition and review process in an effort to streamline the process and reduce the burden on companies submitting proposals. The changes include a reduction in the ATP Proposal Preparation Kit by more than 50 percent; introduction of “rolling submissions”; a staged review process; the electronic submission of proposals, to be piloted later this year. Rolling Submissions The “rolling submission” process allows proposers to submit proposals year round instead of waiting until March (a typical deadline) of the fiscal year to

OCR for page 173
Page 186submit proposals. The other major change is in the proposal submission requirements which provides for a multiple stage and sequential review process. This multiple stage process, as illustrated in Figure 5, is intended to reduce the amount of information required at the time of initial proposal submission. Additional information will be requested at later stages of the process as determinations are made that a proposal has high merit. ATP calls these stages in the review process “gates.” Under this procedure there are four “gates” from proposal submission to award, as follows: Gate 1: Proposer submits detailed information to address the scientific and technological merit selection criterion. Additionally, the proposer submits preliminary information to address the potential for broad-based economic benefits selection criterion. If the information submitted is determined to have high merit ATP notifies the proposer and requests that the required additional information be submitted for consideration in Gate 2. Gate 2: Proposer submits more detailed information to address the potential for broad-based economic benefits selection criterion and detailed budget data. If the information submitted is determined to have high merit, ATP notifies the proposer of its selection as a semi-finalist and the proposal proceeds to Gate 3. ~ enlarge ~ FIGURE 5 Schematic overview of ATP selection process

OCR for page 173
Page 187 Gate 3: The proposer is requested to submit required forms and additional documentation, as necessary, and may be invited to NIST for an oral review. If ATP determines, based on all the information received, that the proposal has sufficiently high merit to be funded, the proposal is considered a finalist and proceeds to Gate 4. Gate 4: Final award processing and issuance, if selected. CONCLUSIONS Steady Refinement The ATP has worked steadily to refine the structures of its competitions and its processes for improved management. The past 10 years have seen many adjustments in the proposal review and project selection process, and additional changes are likely as the program staff and management continue to learn from experience and to adjust to the changing requirements of a program focused on rapidly evolving technologies and companies competing in a global marketplace. Lessons from Evaluation An additional source of information that is increasingly informing the selection process is ATP's evaluation program. As was noted in the April 2000 review of the program by the National Academies, the ATP benefits from one of the most extensive and intensive evaluation efforts of any federal partnership program. This evaluation effort has produced valuable analysis that sheds light on what is working and what is not. Importantly, these results have been incorporated in the ATP decision process primarily through the training of SEB members and independent reviewers. For example, reviewers are informed about what types of problems to look for when joint ventures are proposed and how through questioning to uncover such problems. Identifying Long-term Benefits One important aspect of the training is to look for long-term economic benefits that flow from the project, that is to say, that the value of the project to ATP is evaluated not only on the success of the award recipient but on larger, more diffuse national economic benefits that result from the project. For example, the reviewers look for the use of technology in multiple industries and for the sharing of the technology within the membership of a joint venture. Due to this careful emphasis on long-term economic goals, of the completed projects that have been evaluated to date, 16 percent are demonstrating large returns and approximately 58 percent are demonstrating solid returns.

OCR for page 173
Page 188 The Impact of Scale on Operations Changes in the scale of the ATP have also influenced the choice of operational processes. While requirements in terms of fairness, attention to stakeholder needs, efficiency, and effectiveness have remained constant, changes in operational procedures have occurred. These changes were necessary to allow the capacity of handling in a timely way the number of proposals received in the face of budget-restricted competition cycles. Efforts to achieve processes compatible with larger scale operations, such as implementation of Focused Program Competitions, have in some cases given rise to other problems, such as Congressional objections that the Focused Program selection process was not adequately vetted by outside reviewers in deciding which technologies would be proposed. The Impact of Uncertainty The numbers of proposals received are influenced not only by how the program operates, but also, and primarily, by the perceived certainty or uncertainty of its funding levels. The costs that companies incur when they propose to the ATP are certain. However, the available funding for which the firms compete is often perceived as highly uncertain. The ATP is constantly alert for ways to make the program work better for industry in the face of the substantial obstacle of funding uncertainty and its discouraging effects on industry participation. Further changes in the competition structure are, therefore, likely to be made. How Has This All Worked? A study just being finalized by the Economic Assessment Office of ATP finds that from a portfolio perspective, the results look strong for the ATP. The estimated net benefits attributed to the program from the top performers alone far exceed the $1.640 billion publicly funded portion of the ATP to date, 7 suggesting that the program is on track to produce a high return for the nation. REFERENCES M.P. Feldman and M.R. Kelley. 2001 . “Leveraging Research and Development: The Impact of the Advanced Technology Program.” in this volume. National Institute of Standards and Technology. ATP web site. http://www.atp.nist.gov . 7 Because of the matching requirement from the private participants for ATP awards, the total funding of new technologies from the program is approximately $3.3 billion, of which $1.6 is from industry participants. This sum includes 522 projects awarded to 1162 participants and 1045 subcontractors, and involves 172 joint ventures.