. "Annex I Tasks to Further Develop and Implement the Methodology." An Assessment of the Small Business Innovation Research Program: Project Methodology. Washington, DC: The National Academies Press, 2004.
The following HTML text is provided to enhance online
readability. Many aspects of typography translate only awkwardly to HTML.
Please use the page image
as the authoritative form to ensure accuracy.
An Assessement of the Small Business Innovation Research Program: Project Methodology
organizations/agencies, and probably to program managers within those organizations.5 Another measure of quality specific to SBIR research is the utility of outputs to the funding agency and/or to the market. Information about commercialization will also come from the surveys and case studies.6
Task 3: Collect information and evaluate, using traditional metrics, the economic benefits of the SBIR program.7 Griliches (1958) and Mansfield (1977) pioneered the application of fundamental economic insight to the measurement of private and social rates of return to innovative investments.8 Streams of investment costs generate innovations and associated streams of economic benefits over time. Once identified and measured, these streams of costs and benefits are used to calculate such performance metrics as social rates of return and benefit-to-cost ratios. Thus, the evaluation question that can be answered from this traditional approach is: Given the investment costs and the social benefits, what is the social rate of return from the innovation?
The economic benefits achieved by the SBIR program can be evaluated using several methods, including survey and case study methods. Information collected in Task 1 and Task 2 will underpin the details of the approach.
The evaluation literature and the evaluation experience of the Committee and that of the expert consultants reporting to the Committee,9 suggests that the first-level net benefits will be quantified based on both retrospective and prospective survey data. The information collected in Task 1 and Task 2 should identify relevant first-level output measures such as sales, employment growth, new products and processes, leveraged R&D investments (including additional R&D investment dollars as well as the establishment of new research partnerships10), and enhanced access to capital markets.11 The surveys will also include questions that address management issues.
Second-level beneficiaries from the SBIR program include the agency that funded the project under evaluation. Third-level beneficiaries are the public- and private-sector consumers of the commercialized innovation developed by the award recipient. Both the evaluation literature and the evaluation experience of the Committee and others suggest that second- and third-level benefit data – quantitative and qualitative –can be collected through focused case studies.
Task 3 relates to the second objective of this study. As noted above, part of the Congressional charge to the NRC is to compare the findings from Task 3 to evaluations of similar Federal research and development expenditures. Several Committee members and contract researchers have experience in evaluating Federal research and development programs. At the completion of Task 3, this expertise and experience will be applied to the task of assessing and evaluating the SBIR research results.
For an example of an analysis of NASA SBIR program managers’ qualitative information, see Archibald, R.B. and Finifter, D.H. “Evaluating the NASA Small Business Innovation Research Program: Preliminary Evidence of a Trade-off Between Commercialization and Basic Research,” Research Policy, April 2003.
Information about commercialization will also be collected from funded company officers and individual research scientists in a later task.
The Committee interprets “evaluation” to be a broader analysis than would be undertaken in an “impact assessment.” An impact assessment focuses on the impact (e.g., measured in terms of rates of return or benefit-to-cost comparisons) of the funded research on the agency’s stakeholders (e.g., small businesses). An evaluation includes an impact assessment as well as an examination of the portfolio of research vis-à-vis the objectives of the funding agency and an examination of how well the agency’s funding program are being managed. See Link, A.N. Economic Impact Assessment: Guidelines for Conducting and Interpreting Assessment Studies, Planning Report 96-1, National Institute of Standards and Technology, May 1996, for the application of these important terms as applied within the National Institute of Standards and Technology (NIST). See Georghiou, L., Dale, A., and Cameron, H. Special Issue of Research Evaluation on National Systems for Evaluation of R&D in the European Union, April for an application of these terms as applied within the European Union. As such, preliminary discussions with agencies suggest that a review of commercialization after the award would be useful to them for management purposes. The team anticipates viewing commercialization as an output of research and thus would logically become a part of the evaluation effort in this task.
See Griliches, Z. “Research Costs and Social Returns: Hybrid Corn and Related Innovations,” Journal of Political Economy, 1958. See also Mansfield, E., Rapoport, J., Romeo, A. Wagner, S., and Beardsley, G. “Social and Private Rates of Return from Industrial Innovations,” Quarterly Journal of Economics, 1977.
Some of the team members were involved in the evaluation of the Department of Defense’s Fast Track program. See National Research Council, SBIR: An Assessment of the Department of Defense Fast Track Initiative, 2000, op cit.
See Hagedoorn, J., Link, A.N., and Vonortas, N.S. “Research Partnerships,” Research Policy, April 2000. (2000) for a review of the theoretical and empirical literature related to research partnerships and R&D efficiency.
The Advanced Technology Program (ATP) within the National Institute of Standards and Technology (NIST) has a long and successful history of collecting through surveys such output measures to proxy first-level social benefits. See Ruegg, R.T. “The Advanced Technology Program, Its Evaluation Plan, and Progress in Implementation,” Journal of Technology Transfer, November 1997. See also Ruegg, R.T. and Feller, I. “A Toolkit for Evaluating Public R&D Investments: Models, Methods, and Findings from ATP’s First Decade,” NIST GCR 02-842, National Institute of Standards and Technology, May 2003. Finally, see the research papers contained in National Research Council, The Advanced Technology Program: Assessing Outcomes, C. Wessner (ed.). Washington, D.C.: National Academy Press, 2001.