award was made and later the awardee commercialized a new product. Rather, a goal of the study will be to determine if the commercialization or its timing or some other associated attribute of importance was likely caused by the SBIR award. Evaluation is directed at ruling out alternative, competing explanations of an observed change.72 Additionality tests are usually applied by contrasting the changes that occurred in a “program group” with what, hypothetically, they would have done without the program, or, better, what a comparable group that did not participate in the program actually did relative to the program group. In selecting comparison groups, it is important to ensure that they do not differ in important ways other than participation. Additionality tests can be strengthened by using statistical tools and econometric techniques to help rule out other causes.

The comparison of what program participants would have done differently without the program is usually ascertained by interviews or surveys, using what are called “counterfactual questions.” Counterfactual questions, for example, have been used in a variety of ATP surveys.73 They have also been used in ATP case studies to help estimate project impacts.74

Use of a control group will entail the comparison of a program group with a comparable group that did not participate in the program. Although identifying appropriate control groups will be challenging and can be controversial, the approach is worth considering. Good examples of the use of control groups in evaluation are also available from ATP studies, where they have been used in conjunction with surveys and supporting econometric analysis. 75

Use of other evaluation methods

Special studies may be required that use methods other than surveys and case studies—such as bibliometric or sociometric analysis. Such needs will be determined as the study progresses.

   

underscore the challenge of assessing the impact of public support for private R&D and the need to address the challenges in a comprehensive fashion.

72  

For a further discussion, see R. Ruegg and I. Feller, A Toolkit for Evaluating Public R&D Investments: Models, Methods, and Findings from ATP’s First Decade, NIST GCR 02-842 (Gaithersburg, MD: National Institute of Standards and Technology, May 2003).

73  

See, for example, J. Powell and K. Lellock, Development, Commercialization, and Diffusion of Enabling Technologies: Progress Report, NISTIR 6491 (Gaithersburg, MD: National Institute of Standards and Technology, April 2000).

74  

See A. N. Link, Advanced Technology Program; Early Stage Impacts of the Printed Wiring Board Research Joint Venture, Assessed at Project End, NIST GCR 97-722 (Gaithersburg, MD: National Institute of Standards and Technology, 1997); and Sheila A. Martin, Daniel L. Winfield, Anne E. Kenyon, John R. Farris, Mohan V. Baal, and Tayler H. Bingham, A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies, GCR 97-737 (Gaithersburg, MD: National Institute of Standards and Technology, 1998).

75  

See, for example, Maryann Feldman and Maryellen Kelley, Winning an Award from the Advanced Technology Program: Pursuing R&D Strategies in the Public Interest and Benefiting from a Halo Effect, NISTIR 6577 (Gaithersburg, MD: National Institute of Standards and Technology, 2001).



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement