Page 189

Leveraging Research and Development: The Impact of the Advanced Technology Program

Maryann P. Feldman

Johns Hopkins University

Maryellen R. Kelley

National Institute of Standards and Technology

ABSTRACT

This paper examines the factors that affect a firm's chances of winning an award from the Advanced Technology Program (ATP) and the subsequent impact of the award on a firm's success in raising additional funds for its research and development (R&D) activities. Analysis of data from a survey of 1998 ATP applicants shows that proposals with higher ratings by technical and business/economic experts have a greater chance of winning an award in a process that is regarded by winners and non-winners as fair and rational. Further, the projects and firms selected by ATP are more willing to share their research findings with other firms, and tend to be those that open up new pathways for innovation through combining technical areas or by forming new R&D partnerships. Most of the non-winners have not proceeded with any aspect of the R&D project proposed to ATP and, of those that have, most did so at a smaller scale. Furthermore, the ATP award has prestige value for the winning firms; the halo effect from the award increases the success of these firms in attracting additional funding from other sources. Our conclusion is that the ATP is leveraging activities that have a strong potential for broad-based economic benefit.

Acknowledgements: We thank Wesley Cohen, Bronwyn Hall, Frank Lichtenberg, Jeanne Powell, Rosalie Ruegg, Andrew Wang, and Charles Wessner for their detailed comments on earlier drafts of this paper. We are also grateful for the many helpful comments and advice we received from participants at workshops sponsored by the National Academy of Science, the National Bureau of Economic Research, and at presentations at the annual meeting of the American Economic Association and the Western Economic Association.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 189
Page 189 Leveraging Research and Development: The Impact of the Advanced Technology Program Maryann P. Feldman Johns Hopkins University Maryellen R. Kelley National Institute of Standards and Technology ABSTRACT This paper examines the factors that affect a firm's chances of winning an award from the Advanced Technology Program (ATP) and the subsequent impact of the award on a firm's success in raising additional funds for its research and development (R&D) activities. Analysis of data from a survey of 1998 ATP applicants shows that proposals with higher ratings by technical and business/economic experts have a greater chance of winning an award in a process that is regarded by winners and non-winners as fair and rational. Further, the projects and firms selected by ATP are more willing to share their research findings with other firms, and tend to be those that open up new pathways for innovation through combining technical areas or by forming new R&D partnerships. Most of the non-winners have not proceeded with any aspect of the R&D project proposed to ATP and, of those that have, most did so at a smaller scale. Furthermore, the ATP award has prestige value for the winning firms; the halo effect from the award increases the success of these firms in attracting additional funding from other sources. Our conclusion is that the ATP is leveraging activities that have a strong potential for broad-based economic benefit. Acknowledgements: We thank Wesley Cohen, Bronwyn Hall, Frank Lichtenberg, Jeanne Powell, Rosalie Ruegg, Andrew Wang, and Charles Wessner for their detailed comments on earlier drafts of this paper. We are also grateful for the many helpful comments and advice we received from participants at workshops sponsored by the National Academy of Science, the National Bureau of Economic Research, and at presentations at the annual meeting of the American Economic Association and the Western Economic Association.

OCR for page 189
Page 190 INTRODUCTION Through a number of different agencies, the U.S. federal government provides research funding to universities and colleges, government-owned laboratories, and for-profit enterprises. Support for basic research at universities is widely accepted and the links to technical progress are well established. A common question regarding the public funding of research conducted by private firms is if the public interest, specifically the goals of technical advance and economic growth, is being promoted and how. After all, a private firm is expected to fund its own research when it perceives a technical opportunity in an area of its core competency in an established market with reasonable profit expectations. In some cases, however, private firms may not pursue promising technical opportunities for the following reasons: R&D scientific and technical frontiers are risky and the chances of failure are high. 1 An individual firm may not have the capabilities required to develop the technology. Complex new technologies may require collaboration and information sharing; however, the cost of establishing research and development partnerships and making them work productively may provide disincentives to undertaking the project. Private incentives may not be sufficient to induce a firm to undertake the project in the face of difficulties in appropriating the resulting benefits, i.e., the resulting knowledge may flow to others who receive the revenues from the R&D without sharing the cost. 2 Public-private R&D partnerships provide a policy instrument that may alleviate these concerns and encourage private-sector investment. A government R&D partnership may provide a catalyst for private firms to undertake research which will have broad-based knowledge benefits for other firms and other industries. Government programs may provide a neutral forum for competitors to work together on mutually beneficial research. 3 In addition, government funding may reduce the cost to the firm of establishing working relationships with R&D collaborators and provide an incentive for firms to undertake partnerships 1 A. N. Link et al., “The Economics of Science and Technology, The Journal of Technology Transfer, forthcoming, note that the traditional distinctions between basic and applied research have become meaningless in the commercialization of complex technologies. Firms may identify research areas that require additional fundamental research. 2 Paradoxically, the profit incentive that motivates innovative activity by an individual firm also discourages information sharing and collaborative R&D activities between companies. 3 An example of productive and profitable research undertaken by public-private partnerships is Sematech. See National Research Council, Conflict and Cooperation in National Competition for High-Technology Industry, Washington, D.C.: National Academy Press, 1996, footnote 19.

OCR for page 189
Page 191to blend the firm's capabilities with complementary types of external expertise. Collaboration is also expected to have an additional positive effect on the national economy by facilitating the diffusion of economically relevant knowledge within the private sector, thus further leveraging the public investment. Finally, government R&D funds may reduce the firm's scientific and technical risk sufficiently to bring the project within an acceptable rate of return for private-sector investment. 4 Recently, governments around the world have encouraged public-private partnerships. 5 The long-term goal of these programs is to achieve greater productivity growth in an economy through the creation of knowledge that may become incorporated in industrial processes, products and services. The logic for public investment is that, in the long run, the economic benefits to consumers, other firms, and the larger national economy will exceed the private returns realized by the firm that received the research award, and thus justify the public investment. In the United States, the Advanced Technology Program provides an example of a government R&D program that attempts to encourage broadly applicable technical advance. Since 1990, the ATP has provided selective cost-sharing awards for early stage industry-defined research projects with broad-based economic potential. 6 Rationale for Public-Private R&D Partnering The program mission of the ATP thus depends upon its success in selecting projects with attributes that indicate a high potential to generate substantial economic benefits. Of course, there is no way of guaranteeing, in advance, whether an R&D project actually will result in successful commercialization or if the knowledge created will be valued by other organizations. The ATP's review and 4 The general argument for government funding of certain types of R&D activities, as articulated by Z. Griliches, “The Search for R&D Spillovers,” Scandinavian Journal of Economics, 94(Supplement):29-47, is that such subsidies provide an incentive to firms to undertake high-risk R&D projects in which the public rate of return exceeds the private rate of return. This includes, for example, the case where an industry as a whole may benefit from the development of an enabling technology. Private firms typically use some pre-determined benchmark rate of return known as a hurdle rate. The project will only be acceptable if the expected rate of return is above that benchmark. By reducing the cost of the project, government funding will increase the expected rate of return and may make private companies willing to pursue them. 5 See National Research Council, Conflict and Cooperation, pp. 12-40, for a review of international programs that encourage collaboration. For an introduction and review of the objectives of recent U.S. R&D programs see M. R. Kelley, “From Mission to Commercial Orientation: Perils and Possibilities for Federal Industrial Technology Policy,” Economic Development Quarterly, 11(4):313-328. 6 For a discussion of the legislation and policy issues that led to the establishment of the ATP see C. Hill, “The Advanced Technology Program: Opportunities for Enhancement,” in L. Branscomb and J. Keller, editors, Investing in Innovation: Creating a Research and Innovation Policy, Cambridge, MA: MIT Press, 1998, pp. 143-173.

OCR for page 189
Page 192selection process aims to identify high quality research projects proposed by industry that have clearly delineated pathways through which the projects may achieve broad-based economic benefits. 7 High ratings on the technical and economic dimensions by the ATP's reviewers are a pre-requisite for winning an award. In addition to the assessment of the proposal itself, certain behaviors evident in a firm's R&D strategy may provide an indication of how readily the knowledge generated by the government funded R&D project may diffuse to and benefit other organizations.8 In particular, a firm that is connected to other businesses for assistance in planning and carrying out its R&D, or a firm that has extensive linkages to the university research community, may offer more opportunity for other organizations to learn about the technical advances from the funded project.9 Moreover, some firms attempt to guard access to their research results, while others see an advantage in sharing knowledge and are more willing to disseminate their research findings to other firms.10 Clearly, when a firm openly shares information, the results of its research become known more quickly. Since the ATP wants to encourage the diffusion of new knowledge by industry, it may favor projects where firms are attempting to forge new ties in new collaborative R&D efforts, or projects proposed by firms that are embedded in networks, or of company R&D and commercial linkages and firms that are more willing to share the knowledge that their R&D creates. The first research question 7 See the presentation of Rosalie Ruegg, formerly of the ATP and now with TIA Consulting, panel II of the proceedings of this volume, for a description of the direct and indirect pathways of ATP-project impact. 8 Only when scientific and technical advances become known and are incorporated into new products and processes can ATP expect to see any economic benefits realized from the funded project. 9 Studies of the strategies that managers use to learn about other organization's products, technologies, and business practices emphasize the importance of both formal linkages between firms and the informal network ties among engineers, scientists, and managers employed in different organizations (e.g.: Y. Doz, “The Evolution of Cooperation in Strategic Alliances: Initial Conditions or Learning Processes?” Strategic Management Journal, 17[Summer]:55-83; G. Hamel, “Competition for Competence and Inter-Partner Learning Within International Strategic Alliances,” Strategic Management Journal, 12[Summer]:83-103; A. C. Inkpen, The Management of International Joint Ventures: An Organizational Learning Perspective, London, UK: Routledge, 1995; T. Khanna, R. Gulati, and N. Nohria, “The Dynamics of Learning Alliances: Competition, Cooperation, and Relative Scope,” Strategic Management Journal, 19[3]:193-210; Walter W. Powell, Kenneth W. Koput, and Laurel Smith-Doerr, “Interorganizational Collaboration And The Locus Of Innovation: Networks Of Learning In Biotechnology,” Administrative Science Quarterly, 41[1]:116-145). In general, this literature indicates that cooperation among firms is an important pathway for learning about technical advances in other organizations. More specifically, linkages that involve the sharing of resources, whether technical or financial, become important conduits for knowledge transfer among firms. 10 Paradoxically, in order to gain direct and early access to the knowledge and technologies being developed in other organizations, a firm has to be willing to give some of its own accumulated knowledge and technologies to others. See Julia P. Liebeskind, “Knowledge, Strategy, and the Theory of the Firm,” Strategic Management Journal, 17(Winter):93-107.

OCR for page 189
Page 193we address in this paper concerns how these differences in R&D strategies of firms are related to winning an ATP award. Compared to non-winners, do firms that win an ATP award have R&D strategies with greater potential for diffusing knowledge and incorporating technical advance into commercial applications? To the extent that the ATP is selecting projects in firms with R&D strategies that are more accessible to other researchers and firms, the more likely it is that the knowledge generated by the project will be taken up and used. Projects with such attributes are more likely to yield economic benefits, all other things being equal, that extend beyond the firm directly engaged in the project. Leveraging Private Sector R&D Investment Government funding may also spur additional private investment in R&D activities. 11 For those outside the firm, the value of R&D activities in the early stages of technology development is difficult to assess since there is greater risk and uncertainty. 12 As a result, private investors typically focus on later stage projects. In order to secure private investment, firms must be at a proof of concept phase. Public funding is a means for supporting early stage technology development and moving the project to a point where private investors may be willing to invest. 13 Finally, at any given time, there are a large number of investment opportunities, and it may be difficult for private investors to accurately evaluate potential investments. When information about a company's R&D activity comes from a credible source, such as a government agency with a reputation for scientific integrity as well as programmatic expectations of economic significance, the investment decisions of other organizations may be favorably affected. In light of a large number of potential investment opportunities, the announcement of the ATP award may provide information that certifies an awardee's R&D activities from 11 Previous research suggests that the ways in which government provides funding to the private sector is important. Paul A. David, Bronwyn H. Hall, and Andrew A. Toole, “Is Public R&D a Complement or Substitute for Private R&D? A Review of the Econometric Evidence,” Research Policy, 29:497-529, indicate that the private sector is less likely to increase its own R&D spending when government R&D funding is provided through contract R&D programs in which there is potential for follow-on funding and technology procurement. 12 See the presentation of David Morgenthaler of Morgenthaler Venture Capital, panel I, of the proceedings of this volume. 13 See Maryann P. Feldman, Maryellen R. Kelley, with Joshua Schaff and Gabriel Farkas, “Reinforcing Interactions between the Advanced Technology Program and State Technology Programs. Volume II: How The States Assist High-Tech Start-up Companies,” NISTIR 6523, June 2000, for an analysis of in-depth case studies that found this relationship. These case studies were used to design this statistical study that we report on here.

OCR for page 189
Page 194the perspective of the investment community. 14 Hence, our second research question focuses on how ATP's funding makes a difference to firms in carrying out the type of high-risk, potentially high payout R&D proposed to the program. We address this question in two parts. Do non-winners continue with the proposed research project without ATP funding? Does the ATP award induce additional investment by other organizations in the R&D project? The next section provides an introduction to the ATP and its mission. Section 3 presents the data that we will use to answer our research questions. Section 4 describes how we measure the potential of the project to provide broad-based economic benefits and the basis for our conclusions that ATP is satisfying its mission. Section 5 considers what happened to the non-awarded firms and projects one year later. We conclude that the majority of the non-awardees did not pursue the research. Section 6 considers the effect of the ATP award on the ability to secure additional funding. We conclude that ATP provides a certification function that increases the amount of funds that the firm is able to raise subsequently. This paper concludes by considering the ability of government-funded R&D conducted by private firms to create innovation and potential long-term economic benefits. SELECTING R&DPROJECTS WITH BROAD-BASED BENEFITS: BACKGROUND ON THE ADVANCED TECHNOLOGY PROGRAM ATP depends on the initiative of industry to define the goals for the proposed research projects. 15 All applicants are provided a guide to use in preparing proposals. The guide includes information on the evaluation criteria that the ATP 14 J. Lerner, “The Government as Venture Capitalist: The Long Run Impact of the SBIR Program,” Journal of Business, 72(3):285-318, suggests that the Small Business Innovation Research (SBIR) program award provides a certification function that informs private investors of a notable opportunity. 15 The ATP is fundamentally different from other government R&D programs as the focus is industry-defined. This contrasts with the SBIR as a case in point. The SBIR program is a set aside of U.S. Federal Agency funds specifically earmarked for small business. While the program has demonstrated positive effects on the development of new technologies and the growth of the companies, the areas of interest are defined by the agency, not the participating companies. For an overview of the SBIR program and its challenges see National Research Council, The Small Business Innovation Research Program: Challenges and Opportunities, Charles W. Wessner, editor, Washington, D.C.: National Academy Press, 1999; and for a review of the operation of SBIR at the Department of Defense see National Research Council, The Small Business Innovation Research Program: A Review of the Department of Defense Fast Track Initiative, Charles W. Wessner, editor, Washington, D.C.: National Academy Press, 2000.

OCR for page 189
Page 195employs in selecting awardees. To merit funding, the project must have both scientific and economic potential: The research must be challenging, with high technical risk… aimed at overcoming an important problem(s) or exploiting a promising opportunity…[and] must have a strong potential for advancing the state of the art and contributing significantly to the U.S. scientific and technical knowledge base. The proposed technology must have a strong potential to generate substantial benefits to the nation that extend significantly beyond the direct returns to the proposing organization(s). The proposal must explain why ATP support is needed and what difference ATP funding is expected to make. The pathways to economic benefit…includ[e] the proposer's plan for getting the technology into commercial use, as well as additional routes that might be taken to achieve broader diffusion of the technology. (Advanced Technology Program Proposal Preparation Kit, 1999: 7-8). Independent technical experts and specialists in the business planning evaluate every proposal. The selection process involves an assessment of the project's contribution to technical advance, its economic potential, and the need for public funding. In any given year, fewer than 20 percent of proposed projects have actually been awarded funding and the overall average is 12%. STUDY OFTHE 1998 ATP APPLICANTS: DATA DESCRIPTION To assess the ATP program we focus on the competition that was held during the summer of 1998. A total of 502 proposals were submitted. Because ATP guidelines encourage joint projects, there were a total of 822 organizations that applied in this competition. Our primary interest is the 741 for-profit enterprises that applied to ATP in 1998. 16 This group of firms constitutes our sampling frame. The award winners were announced in October 1998. ATP funded 80 proposals that involved 161 organizations, including 147 forprofit enterprises. Table 1 shows the distribution of firms by award status and firm size. About 20 percent of all applicant firms received an ATP award in 1998. Small firms, employing fewer than 500 employees, constituted nearly 70 percent of all applicants. Although firms in this size category were only 52 percent of ATP award recipients, they accounted for 83 percent of all awards that were made to individual companies in that year. There were also 27 joint venture (JV) projects funded in 1998. These JV projects include 65 percent of all awardees and 87 percent of the firms with more than 500 employees. 16 Private firms must lead ATP projects but there is frequent collaboration with for-profit firms. In 1998 there were 38 universities, 29 non-profit organizations, seven government labs, and seven foreign-owned U.S. subsidiaries.

OCR for page 189
Page 196 TABLE 1 Distribution of Applicants by Award Status and Firm Size Award Status Award Winners Non-Winners All Applicants Percent of Firms ≥ 500 employees 48% 27% 31% Percent of Firms < 500 employees 52% 73% 69% Number of Firms 147 (20%) 594 (80%) 741 To collect data on the 1998 applicants, we conducted a survey on the ATP application process, and the experiences after the ATP application. 17 All interviews were completed over a six-month period (June-December 1999). Our respondent was the person identified as the technical lead for the proposed project. If this person was no longer with the company, we asked to speak to the individual who was most knowledgeable about the proposal and the company's R&D activities in the area identified in the proposal. 18 The telephone questionnaire required 20-30 minutes to complete. 19 Our sample consisted of 100 percent of the winning firms and a simple random sample of 50 percent of the non-winners. Thus, we contacted 297 non-winners and 147 awardees one year after the 1998 ATP selection process. We completed interviews for 119 winners for an 81 percent response rate (119/147). For the non-winners, we discovered that within one year there were 49 cases that we could not interview, either because the company was no longer in business (23 cases) or because the person responsible for preparing the ATP proposal was no longer employed at the company and the company was not pursuing any aspect of the R&D proposed to ATP (26 cases). We adjusted our response rate accordingly. We completed interviews for 122 non-winners, for a 50 percent response rate (122/[297-49]). This yields an overall effective response rate of 61 percent. 17 The survey instrument was reviewed and approved by the Office of Management and Budget and the JHU Committee on the Use of Human Subjects. OMB granted approval (no. 0693-0027) for Johns Hopkins University to conduct the survey on 24 March 1999. 18 Prior to calling our respondents to conduct the interviews, we followed standard survey method procedures, sending a letter to all potential respondents in the selected sample to explain the purpose of the survey, identify ATP as the sponsor of the study, and asked for cooperation. This letter also contained a statement of confidentiality and assurance that responses to any of the survey questions would remain anonymous, and would not be publicly released in any form that would identify a specific individual or company. The letter included a list of questions that the respondent might find helpful to have in advance of the telephone interview. In addition, our mailing included an introductory letter from the ATP. The Schaefer Center for Survey Research at the University of Baltimore conducted the telephone interviews. 19 A copy of the questionnaire is available from the authors upon request.

OCR for page 189
Page 197 The survey results were matched with company and project-specific data from other sources. First, we used independent sources such as the CorpTech Database and Hoovers Online Company and Industry Network to verify survey responses concerning employment, financing, and the founding date of the company. 20 Second, we matched each record with information from ATP administrative records on the technology area of the proposal, the results of the ATP proposal review process, and the firm's prior history of applications and awards. PATHWAYS TO INNOVATION: VALUATING THE POTENTIAL FOR BROAD-BASED TECHNICAL ADVANCESAND ECONOMIC GROWTH A project may have greater economic potential if the firm embarks on a new technical problem or forms a new R&D collaboration. Although R&D collaborations are widely recognized as an important strategy for learning about technical advance, 21 establishing new collaborative relationships is especially difficult and costly for firms. 22 Table 2 shows the percent of projects proposed to ATP in 1998 that involved two or more organizations as research partners, and the percent of new partnerships and new technical areas proposed by ATP award winners compared to non-winners. Seventy-nine percent of the 1998 ATP applicants in our sample included other organizations in their proposal. There is no difference between award winners and non-winners in their propensity to have partners. However, the percent of award winners identifying a new collaborator, as their most important research partner is much higher compared to non-winners (59 percent versus 42 percent). Award winners were also more likely to propose a project in a technical area new to the firm (47 percent versus 19 percent). One firm's R&D activities may have the potential to augment other firms' innovative capabilities but only if research results and knowledge are shared. Competitors' reluctance to collaborate and share research results may be likely, however, for certain complex technical areas sharing information may be essential for firm's technical advance. Moreover, an award winning firm's willingness to diffuse research results affords a greater opportunity for other firms to benefit from government funded research. As indicated in Table 3, we used three statements about the company's approach to construct a measure of the firms' tendency towards either openness or secrecy in conducting its own R&D. The possible scores ranged from 0 to 3. We 20 See http://www.corptech.com/ and http://www.hoovers.com/ . 21 For recent research on the advantages of collaboration see Doz, “The Evolution of Cooperation”; Hamel, “Competition for Competence”; Inkpen, The Management of International Joint Ventures; Khanna, Gulati, and Nohria, “The Dynamics of Learning Alliances”; Powell, Koput, and Smith-Doerr, “Interorganizational Collaboration.” 22 K. R. Harrigan, “Joint Ventures and Competitive Strategy,” Strategic Management Journal, 9(March-April):141-158.

OCR for page 189
Page 198 TABLE 2 Indicators of the Creation of New Pathways to Innovation New Partnerships Percent of 1998 applicants who included other organizations in the ATP proposal 79% If Yes, was this a new partnership? Award winners 59% Non-winners 42% New Technical Area Percent of applicants proposing a project for which it did not have a prior plan—a technology new to the company: Award winners 47% Non-winners 19% TABLE 3 Tendency Towards Openness or Secrecy Willingness to make research results available to other firms: To what extent do you intend to make your research results available to other firms and industries? 1 = almost always or sometimes; 0 = rarely or never Do you think that keeping your company's R&D knowledge from spreading to other firms is important to your firm's long run success? 1 = no; 0 = yes Would you ever consider not engaging in new R&D activity because you believe another firm may benefit from it? 1 = no; 0 = yes expect that most firms will have a tendency to secrecy. 23 Firms that answered yes to two of the three questions were considered to exhibit a willingness to share research results. The potential for contributing to scientific and technical advance may be enhanced by a firm's linkages to universities and other institutions. For example, universities are an important source of new knowledge that may be applied to a broad range of industrial problems (Mowery and Rosenberg, 1989). In carrying out R&D a firm may draw upon it connections to university faculty, graduates, 23 For a discussion of the importance of secrecy in R&D strategies see R. R. Nelson, “Capitalism as an Engine of Innovation” in The Sources of Economic Growth, Richard R. Nelson, editor, Cambridge and London: Harvard University Press, 1996. pp. 52-83., and Julia P. Liebeskind, “Keeping Organizational Secrets: Protective Institutional Mechanisms and their Costs,” Industrial and Corporate Change, 6(3):623-663.

OCR for page 189
Page 199laboratory facilities, and intellectual property to augment its internal capabilities. These relationships are also potential pathways for reciprocal knowledge flows from the firm to the university-based research community and then perhaps on to other firms. The more connections that a firm has to university resources, the greater the potential for knowledge flows in both directions. Our survey used twelve indicators of the firm's connections to universities (see Table 4). 24 Included are universities as R&D partners in general and as resources for the ATP project specifically. The combination of the two sets of measures serves as an index of the strength and diversity of connections to the university-based research community. The ATP mandate stresses commercial and economic benefits as program goals; however, the program does not fund product development or market research. As a consequence, the potential of a technology to attract non-ATP resources for commercialization is critical. This could take the form of additional private capital attracted by the ATP awarded company, or might include resources provided by other companies who take up the technology and incorporate it into their commercial efforts. Connections to other firms, in the planning and development of the project, and more generally, as sources of financial and technical support, may be especially important to the commercialization potential and eventual impact of the project on the economy. We asked about applicant's connections to other firms in preparing the proposal for ATP, as potential collaborators on the project, and more generally, in providing technical and financial resources to the applicant in the two years prior to the application. We used the 19 questions shown in Table 5 to construct a business linkage index. A higher number of links to other businesses signals connection to a diversity of resources particularly important for companies seeking to commercialize their technologies. ATP Awards Risky Projects with the Potential for Broad-based Economic Benefits We find that the ATP awarded those companies who proposed projects involving the formation of new partnerships and new areas of research for the firm. When compared to unsuccessful applicants, the firms that were awarded by ATP were more likely to have extensive linkages to other businesses and to profess a greater willingness to share information about their research findings with other firms. This indicates that they had both the linkages and the inclination so that broad-based economic benefits might be realized. Surprisingly, we find no statistical difference between winners and non-winners in the extent of their linkages to universities. 25 That suggests that the 1998 ATP award winners do not differ, on average, from non-winning applicants, on the extent of their ties to the university-based research community. Given the types of projects that were proposed to the ATP, it is perhaps not surprising that we find that all applicants had links to universities. 24 See Feldman and Kelley, “The Case for Government R&D Additionally,” for further discussion. 25 The average applicant scored 5.45 on the university linkage index with no statistical difference between award winners and non-winners.

OCR for page 189
Page 200 TABLE 4 Questionnaire Items in University Linkages Index UNIVERSITY LINKAGES INDEX Number of Connections (Number of ‘yes' answers to the following questions) For ATP project and proposal: 1. Did your company first learn about ATP from someone at a university? 2. Did a university help you identify the research partner you consider to be the most important for the project you proposed to ATP? 3. In preparing the technical plan portion of your proposal, did you get assistance from someone at a university? 4. In preparing the business plan portion of your proposal, did you get assistance from someone at a university? 5. [If technical lead on the ATP project has been employed with the company less than 5 years], was this person previously employed at a university? General R&D connections to university resources: 6. Does your company have any contracts or licensing agreements for intellectual property at universities? In the two years prior to your ATP application have you used assistance from a university program 7. to address a technical problem? 8. to prepare a business or marketing plan? 9. to recruit R&D employees? 10. In the two years prior to your ATP application have you formed an alliance with a university to address your needs for equipment and facilities? 11. In the two years prior to your ATP application have any of your R&D personnel attended training or technical programs sponsored by a university? 12. In the two years prior to your ATP application, for your R&D or technology development activities, has your company received funds from a university program? We expect that proposals that receive a high rating by technical and business/ economic reviewers to have a greater chance of winning an award. Our results indicate that on both dimensions, the higher the rating the proposal received by reviewers, the greater the chances of the project winning an award. However, having a high quality proposal was not the only criteria. We find that the following attributes increase the likelihood of winning an ATP award: riskier research projects that are new to the firm;

OCR for page 189
Page 201 TABLE 5 Questionnaire Items in Business Linkages Index BUSINESS LINKAGES INDEX Number of Connections (number of ‘yes' answers to the following questions) For ATP project and proposal: 1. Did your company first learn about ATP from: a consulting firm, someone at another company, or a venture capitalist? 2. Did a venture capitalist help you identify the research partner you consider to be the most important for the project you proposed to ATP? 3. In preparing the technical plan portion of your proposal, did you get assistance from a consulting firm? 4. In preparing the technical plan portion of your proposal, did you get assistance from someone at another company? 5. In preparing the business plan portion of your proposal, did you get assistance from a consulting firm? 6. In preparing the business plan portion of your proposal, did you get assistance from someone at another company? 7. [If technical lead on the ATP project has been employed with the company less than 5 years], was this person previously employed at another company? Other Business Ties: In the two years prior to your ATP application have you used assistance from a private venture capital firm 8. to address a technical problem? 9. to prepare a business or marketing plan? In the two years prior to your ATP application, has your company received financing for your R&D or technology development activities from 10. a private venture capital fund? 11. an individual (angel) investor? 12. another company? In the two years prior to your ATP application, to address your needs for equipment and facilities, has your company used 13. private investor or angel financing? 14. venture capital financing? 15. secured bank financing? 16. an alliance with another company? In the two years prior to your ATP application have you used assistance from a private consulting firm 17. to address a technical problem? 18. to prepare a business or marketing plan? 19. In the two years prior to your ATP application have you used assistance from another company to address a technical problem?

OCR for page 189
Page 202 research projects that form new R&D partnerships between organizations; firms that demonstrate a tendency towards openness in communications about research with other firms and institutions; and firms that have a more extensive set of business-to-business linkages. These attributes distinguish award winners from other applicants. Our model included controls for whether the applicant had applied to ATP in the past, the number of times the applicant won an ATP award, the technical area of the proposed R&D project and the relative cost, as a measure of effort, in preparing an ATP proposal. We might expect that companies that have applied or won previously might have an advantage over applicants that have no prior experience with ATP. In addition, the cost of preparing a proposal ranged from $2,000 to $300,000 with a median of $15,000 per firm and we might expect that a professionally presented proposal might affect award status. Our regression results do not find any support that experience with the ATP matters nor do we show that proposal preparation cost is related to winning, independent of other attributes of the project. We conclude that ATP is selecting projects with greater potential for contributing to other firms' R&D efforts and technology use. Award winning projects and firms have attributes that suggest a greater chance of contributing knowledge to other organizations and thus advancing the technical frontier. Moreover, awardwinning firms proposing these projects have greater potential for achieving an economic impact through connections to the resources of other firms and their potential support of the further development and deployment of the technologies. ONE YEAR LATER: WHAT HAPPENED TO NON-WINNERS AND THE R&D PROJECTS THEY PROPOSED TO THE ATP? More than a year passed between the ATP proposal submission and the time we contacted firms to conduct our survey. During the review and selection process, the ATP attempts to determine whether or not the R&D activity that is being proposed is likely to be carried forward at the same speed and scale by the company without any assistance from the ATP. Hence, if ATP attracts proposals that companies would proceed with anyway at the same funding level, with the same timeframe and goals, we would expect to see a high proportion of non-winning projects proceeding during this period of time. An important consideration in the proposal evaluation decision is an assessment of whether the project is likely to proceed without government funding. However, as Table 6 indicates, that is not the case. If a project is believed to be of central importance to a firm that has already funded this type of R&D from its own resources, the ATP is likely to reject the proposal. Similarly, if a company has access to other funding sources (within or outside the firm), the ATP may choose to award its scarce resources to firms that lack access to such funding sources.

OCR for page 189
Page 203 TABLE 6 The Extent to Which Non-Winners Pursue the ATP Proposed R&D Project Without ATP Funding Did not proceed with the project, at any scale 62% Began project on a much smaller scale than proposed to ATP 17% Began project on a somewhat smaller scale than proposed to ATP 12% Began project at about the same scale as proposed to ATP 5% Began project on a somewhat larger scale than proposed to ATP 3% Began project on a much larger scale than proposed to ATP 1% Number of Cases 168 Note: Three respondents were unable or refused to answer this question. More than three-fifths of the non-winners (62 percent) have not proceeded with any aspect of the R&D project that they proposed to ATP. This number includes the non-winners that we discovered had gone out of business in the past year. Also included are the projects where the persons responsible for preparing the proposal no longer worked for the company and there was no one whom we could identify who knew about the proposal or any continuation of that work in the same technical area. 26 In addition to these 49 cases, there were another 66 non-winners who indicated that, in the past year, their company had not proceeded with any aspect of the project proposed to ATP. Nearly 37 percent of the non-awardees began work on the proposed project at some level of effort. However, in most instances (77 percent), the project was pursued at a smaller scale than the company had proposed to the ATP. Only five percent of the firms that received no funding from ATP were proceeding at the same scale as they had originally proposed the previous year, while less than four percent of the non-winners were proceeding with a larger scale effort than had been proposed to ATP. These results suggest that, for the most part, ATP is attracting applicants that need support in order to proceed with their R&D plans. A small minority of non-winners pursued its R&D plans at the same or an even greater scale than originally proposed to ATP. It seems reasonable to assume that the few non-winners who proceeded at the same or greater scale than they had proposed to the ATP were rejected because the ATP saw no need to fund these projects: the projects were promising but the need for public funding was not apparent. 26 In most respects, the projects for firms that either went out of business or where the P.I. (Principal Investigator) was no longer employed at the firm were similar to the other non-awardees. However, we found one important difference. When compared to the evaluation of their business plans, the defunct firms were rated as having stronger, on average, technical plans. This suggests that they were relatively weaker on the business front.

OCR for page 189
Page 204 HALO EFFECT: DOES THE ATP AWARD INCREASE R&D FUNDING FROM OTHER SOURCES? When an enterprise first wins an R&D contract from a government agency the award may be a signal to non-government sources of funding, such as banks, venture capital firms and other potential investors, that the firm has a potential future stream of revenue from a reliable customer (the U.S. government). The ATP awards differ from the usual government R&D contracts to industry, and hence the economic information conveyed by the award is quite different. There is no promise of follow-on funding by the agency. The ATP seeks to fund R&D projects that have commercial applications, and there is no procurement connection—the ATP does not purchase the technology it helps to develop through its awards. In light of the unique features of the program, we assess the potential influence of an ATP award on the subsequent behavior of the investment community to arise from the information the award signifies about both the technical and economic potential of the project and firm. For an ATP award to influence the subsequent decision of other sources of research funding implies that the announcement of the award itself provides valuable and credible information to the investment community. The host agency of the ATP is the National Institute of Standards and Technology (NIST), which is widely recognized to be an important source of technical information and expertise in a number of areas. Yet in order for the ATP to benefit from the prestige of NIST, and for the awards to signify quality or merit, the selection process itself must be perceived as fair and rational (i.e., explainable in relation to the criteria). Our survey provides evidence that the ATP selection and award process is widely viewed as fair and rational. Applicants' Perceptions of the Selection and Award Process The ATP emphasizes the impartiality of its treatment of proposals during the review and selection process. We asked award winners and non-winners alike about their perceptions of the selection process and whether the respondent believed that his/her company would consider applying to ATP in the future. Table 7 and Table 8 present the responses to these questions by the award status of the respondent. As we would expect, Table 7 shows that a high percent (95 percent) of those that won an ATP award perceived the selection and review process to be fair. TABLE 7 Overall, Regardless of the Outcome for Your Proposal, Did the ATP Review and Selection Process Appear Fair? Award Status Award Winners Non-Winners All Applicants Yes 95% 67% 81% No 5% 33% 19%

OCR for page 189
Page 205 TABLE 8 To Your Knowledge, Does Your Company Plan to Apply to the ATP in the Future? Award Status Award Winners Non-Winners All Applicants Definitely/Very likely 82.0% 59.0% 70.0% Undecided 15.0% 12.0% 13.0% Not very likely/Definitely not 4.0% 29.0% 17.0% However, among non-winners, a substantial majority (67 percent) also considered the review and selection process to be fair. These responses suggest that ATP has a reputation for fairness that is widely recognized. As further evidence, consider the responses to the question about the respondent company's plans to apply to ATP in the future shown in Table 8. Although there is a higher negative response from non-winners, a majority (59 percent) is very positive about the prospect of applying to ATP again. 27 When a company fails to win an award, the ATP provides the opportunity for these companies to discuss with panel representatives both the strengths and weaknesses that were identified in the proposal during the review process. Once a company, or group of companies, has been notified that the project was not selected for funding, its management may request to schedule a telephone debriefing session with ATP within a few weeks of the decision. Although there are usually two representatives from the ATP's expert panels (a business and a technical expert), as many individuals from the project as the companies wish may participate in the debriefing. Table 9 shows that the majority of non-winners who responded to our survey, 63 percent, indicated that they had participated in a debriefing in 1998. In addition, we include their assessments of the value of the feedback they received from their discussions with ATP staff during the session. In general, most non- TABLE 9 Percent of Non-Winners that Participated in a Debriefing and Respondents' Assessment of ATP's Helpfulness to the Company Percent of non-winners who participated in a debriefing from ATP staff 63.1 How helpful did you find the debriefing session to be? Percent “yes” very helpful 31.9 reasonably helpful 37.0 not sure 2.6 not particularly helpful 19.4 not at all helpful 9.1 27 Note that the percent of winners and non-winners that are undecided is about the same (15 percent of winners compared to 12 percent of non-winners).

OCR for page 189
Page 206 awardees (69 percent) found the debriefing either to be very helpful (32 percent) or reasonably helpful (37 percent). These responses suggest that most non-winners perceived the ATP staff's explanation for rejecting the proposal to be rational and to provide useful guidance for improving the firm's technical and/or business planning. Pursuing R&D Funding from Other Sources: The Impact of the ATP Award The highly selective and competitive nature of the ATP award may signal potential investors about the quality of the firms' R&D project. As a result, other investors such as private angels or venture capital, other businesses, as well as state governments and other federal agencies, may invest in the firm. We may expect the ATP awards to confer a halo effect to winning firms and projects, boosting the chances of a firm's success in attracting additional funding for its R&D activities. To test for this effect, we considered the group of applicants that pursued other funding sources in the year following the ATP application. Table 10 compares the percentages of firms in the two award status categories that sought additional funding from other sources and the percentages that actually succeeded in attracting funding from these sources. Although a minority of all applicants sought funding from other sources, the percentage of ATP award winners seeking additional funding was much lower. On average, however, the ATP awardees who sought funding from other sources were much more successful in attracting funds than non-winners. We asked firms about the amount of funding that they received for the ATP-proposed research project from non-ATP sources, including private individuals, venture capitalists, other firms, state and local government programs, and other federal government agencies. TABLE 10 Percentage of Firms that Applied to Other Funding Sources to Support Their R&D Activities and Their Success Rates by ATP Award Status Award Status Award Winners Non-Winners All Applicants Percent of all firms in award category that applied to other funding sources 25 47 38 Percent of those firms seeking funding that succeeded in attracting additional investment for their R&D activities 73 33 44

OCR for page 189
Page 207 The ATP Halo Effect In order to determine if the ATP award has the hypothesized halo effect and how much it might be worth in additional funds, we took into account other factors related to winning an ATP award that may also influence other investors. 28 These include the firm's prior success in raising funds, the size of the firm, and the maximum rating by the ATP reviewers. The reviewers' ratings serve as a measure of the technical quality and promise of a project and of the business and economic potential of the technology. We are interested in determining the influence of the ATP award as a pure signal to the investment community, and our statistical analyses confirm our hypothesis. After we have controlled for these other factors, we still find a significant halo effect from an ATP award. 29 Award winners who seek additional funding from non-ATP sources were more successful than non-winners and received a larger amount of funding. Our analysis concludes that winning an ATP award significantly increases the firm's success in attracting additional funds from other sources for R&D activities. Our findings provide strong evidence that the ATP award confers a halo effect on winners that makes them more likely to attract other funding when compared to non-winners of the same size, and age with project of similar business and technical quality. Thus, our results confirm that the ATP award appears to send a market signal that certifies that the firm and the technology are promising. LEVERAGING R&D: THE POTENTIAL OF THE ATP Growth theorists describe two roles for government R&D policy. 30 The first role is in funding the supply of R&D. Most research to date has focused on this topic. The second role for government is providing incentives for private sector R&D and generating a set of behavioral responses that would be expected to stimulate R&D and promote innovation and economic growth. We know the most about the supply of R&D resources. Less is known about the ways in which government programs provide incentives for activities that generate innovation and economic growth. In this paper, we have shown how a government program is leveraging private sector R&D activities by selecting projects with greater potential for diffus 28 We used a TOBIT model to estimate the total amount of funding a firm succeeded in raising from a number of different sources. The TOBIT model accommodates the truncated nature of the dependent variable that has a lower bound of zero. See Feldman and Kelley, “The Case for Government R&D Additionally,” for details. 29 Not surprising, we find that prior success in raising funds matters. The greater the amount of funding a firm was able to raise in the past, the more success it will have in obtaining additional funding. We also find that smaller firms were more successful in raising additional funds. 30 Boyan Jovanovic, “Growth Theory,” NBER Working Paper no. 7468, 2000.

OCR for page 189
Page 208ing knowledge and for contributing to the development of new technologies for commercial use. In essence, the mission of the Advanced Technology Program is to support private sector R&D projects that offer potential for contributing to technical advance and for realizing economic value. Our findings suggest that the operation of the program is consistent with its mandate and objectives. While the program is selecting projects that are rated highly by independent technical and business experts, it awards technically risky, new projects proposed by firms that have strong external linkages to other institutions and exhibit a willingness to diffuse knowledge based on their research findings. We conclude that ATP is selecting projects and firms that have greater potential for increasing the circulation of new knowledge and for having the business connections necessary to realize economic benefits from its activities. The extent of a firm's linkages to other enterprises and to the resources of universities indicate that a firm is well-positioned to both tap the capabilities of other actors in our nation's innovation system and be a conduit for carrying the knowledge generated by an ATP project to use by other firms. These types of connections are important to the overall operation of the entire U.S. innovation system. We also find that the ATP supports the formation of new linkages and the initiation of R&D projects that bridge different technical areas, opening up new pathways to innovation. We provide evidence that the investment community values the ATP award. Among firms that seek additional funding, we find that ATP award winners are more successful than non-winners. The halo effect from the ATP award is independent of the ratings of the quality of the project, and the firm's prior success in winning funds from these sources. Since few R&D projects proposed to ATP actually proceed at a comparable level, we conclude that the ATP is stimulating additional investment in risky R&D projects that would otherwise not be funded by the firms themselves or other funding sources. REFERENCES Cohen, Linda R. and Roger G. Noll. 1991 . The Technology Pork Barrel . Washington, DC : The Brookings Institution . Cohen, Wesley M. and Daniel A. Levinthal. 1990 . “Absorptive Capacity: A New Perspective on Learning And Innovation.” Administrative Science Quarterly 35(1): 128-152 . David, Paul A., Bronwyn H. Hall, and Andrew A. Toole. 2000 . “Is Public R&D a Complement or Substitute for Private R&D? A Review of the Econometric Evidence.” Research Policy 29(4-5): 497-529 . Doz, Y. 1996 . “The Evolution of Cooperation in Strategic Alliances: Initial Conditions or Learning Processes? Strategic Management Journal 17(Summer): 55-83 . Feldman, Maryann P. and Maryellen R. Kelley. 2000 . “The Case for Government R&D Additionally: An Investigation of the Advanced Technology Program Selection Process.” Paper presented at the National Bureau of Economic Research Summer Institute session on Productivity, Science and R&D.

OCR for page 189
Page 209 Feldman, Maryann P., Maryellen R. Kelley, with Joshua Schaff and Gabriel Farkas. 2000 . “Reinforcing Interactions between the Advanced Technology Program and State Technology Programs. Volume II: How The States Assist High-Tech Start-up Companies.” NISTIR 6523. June. Flamm, Kenneth. 1988 . Creating the Computer: Government, Industry, and High Technology . Washington, D.C. : Brookings Institution . Gittelman, Michelle and Bruce Kogut. 2000 . “Why Do Firms Publish Their Research (By Their Own Scientists?): Publication and Patenting by U.S. Biotechnology Firms.” Paper presented at the Western Economic Association meeting in Vancouver, BC. July 1. Griliches, Z. 1992 . “The Search for R&D Spillovers.” Scandinavian Journal of Economics 94(Supplement): 29-47 . Guellec, D. and B. von Pottelsbughe. 2000 . “The Impact of Public R&D Expenditures in Business R&D.” Paper presented to NBER Summer Institute. July 25. Hagedoorn, John, Albert N. Link, and Nicholas S. Vonortas. 2000 . “Research Partnerships.” Research Policy 29(4-5): 567-586 . Hall, Bronwyn H., Albert N. Link, and John T. Scott. Forthcoming. Universities as Research Partners in ATP-Funded Projects . Gaithersburg, MD : National Institute for Standards and Technology . Hamel, G. 1991 . “Competition for Competence and Inter-Partner Learning Within International Strategic Alliances.” Strategic Management Journal 12(Summer): 83-103 . Harrigan, Kathryn R. 1988 . “Joint Ventures and Competitive Strategy.” Strategic Management Journal 9(March-April): 141-158 . Hill, Christopher T. 1998 . “The Advanced Technology Program: Opportunities for Advancement” in Investing in Innovation: Creating a Research and Innovation Policy that Works . Lewis Branscomb and James Keller, editors. Cambridge, MA : The MIT Press . Inkpen, A. C. 1995 . The Management of International Joint Ventures: An Organizational Learning Perspective . London : Routledge . Jaffe, Adam B. 1996 . Economic Analysis of Research Spillovers: Implications for the Advanced Technology Program . GCR 97-708. December. Jovanovic, Boyan. 2000 . “Growth Theory.” NBER Working Paper 7468. Kelley, Maryellen R. 1997 . “From Mission to Commercial Orientation: Perils and Possibilities for Federal Industrial Technology Policy.” Economic Development Quarterly 11 (4): 313-328 . Khanna, T., R. Gulati, and N. Nohria. 1998 . “The Dynamics of Learning Alliances: Competition, Cooperation, and Relative Scope.” Strategic Management Journal . 19(3): 193-210 . Klette, Tor Jakob, Jarle Moen, and Zvi Griliches. 2000 . “Do Subsidies to Commercial R&D Reduce Market Failures? Microeconomic Evaluation Studies.” Research Policy . 29(4-5): 471-495 . Lerner, Josh. 1999 . “The Government as Venture Capitalist: The Long Run Impact of the SBIR Program.” Journal of Business 72(3): 285-318 . Liebeskind, Julia P. 1996 . “Knowledge, Strategy and the Theory of the Firm.” Strategic Management Journal 17(Winter): 93-107 . Liebeskind, Julia P. 1997 . “Keeping Organizational Secrets: Protective Institutional Mechanisms and their Costs.” Industrial and Corporate Change 6(3): 623-663 . Link, Albert N., David B. Audretsch, Maryann P. Feldman, John E. Jankowski, Donald Siegel, Paula Stephan, Gregory Tassey, and Charles Wessner. Forthcoming. “The Economics of Science and Technology.” Journal of Technology Transfer . Mowery, David and Nathan Rosenberg. 1989 . Technology and the Pursuit of Economic Growth . Cambridge : Cambridge University Press . Narayanan, V. K., G. E. Pinches, J. M. Kelm, and D. M. Lander. 2000 . “The Influence of Voluntarily Disclosed Qualitative Information.” Strategic Management Journal 21(7): 707-722 . National Research Council. 1996 . Conflict and Cooperation in National Competition for High-Tech nology Industry: A Cooperative Project of the Hamburg Institute for Economic Research, Kiel Institute for World Economics and the National Research Council on “Sources of International Friction and Cooperation in High-Technology Development and Trade.” Washington, DC : National Academy Press .

OCR for page 189
Page 210 National Research Council. 1999 . The Small Business Innovation Research Program: Challenges and Opportunities . Charles W. Wessner, editor. Washington, D.C. : National Academy Press . National Research Council. 2000 . The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative . Charles W. Wessner, editor. Washington, D.C. : National Academy Press . National Science Board. 1998 . Science & Engineering Indicators—1998 . Washington, DC : U.S. Government Printing Office . Nelson, Richard R. 1992 . “What Is ‘Commercial' and What Is ‘Public' about Technology, and What Should Be?” in Technology and the Wealth of Nations . Nathan Rosenberg, Ralph Landau and David Mowery, editors. Stanford : Stanford University Press , pp. 57-71 . Nelson, Richard R. 1996 . “Capitalism as an Engine of Innovation” in The Sources of Economic Growth . Richard R. Nelson, editor. Cambridge and London : Harvard University Press , pp. 52-83 . Powell, Walter W., Kenneth W. Koput, and Laurel Smith-Doerr. 1996 . “Interorganizational Collaboration And The Locus Of Innovation: Networks Of Learning In Biotechnology.” Administrative Science Quarterly 41(1): 116-145 . Sako, Mari. 1992 . Prices, Quality and Trust: Inter-firm Relations in Britain & Japan . New York : Cambridge University Press . Sako, Mari. 1994 . “Supplier Relationships and Innovation” in The Handbook of Industrial Innovation . Mark Dodgson and Roy Rothwell, editors. Cheltenham, UK and Brookfield, MA : Elgar . Teece, David J. 1992 . “Competition, Cooperation and Innovation: Organizational Arrangements for Regimes of Rapid Technological Progress.” Journal of Economic Behavior and Organization 18: 1-25 . Trajtenberg, Manuel. Forthcoming. “R&D Policy in Israel: An Overview and Reassessment” in Technology Policy for the Knowledge-Based Economy . M. P. Feldman and A. N. Link, editors. Boston, MA : Kluwer Academic Publishers . von Hippel, Eric. 1988 . The Sources of Innovation . New York : Oxford University Press . Wallsten, Scott J. 1998 . “Rethinking the Small Business Innovation Research Program” in Investing in Innovation: Creating a Research and Innovation Policy that Works . Lewis Branscomb and James Keller, editors. Cambridge : The MIT Press .