The NIST ATP has developed a complex program-evaluation tool, the business reporting system (BRS). The BRS, which was implemented in 1994, is used to track companies that have received funding through the ATP. It is an impressive evaluation tool that comprehensively evaluates the business and economic impacts of each research project from start to finish (see Box C-1). Companies are asked to respond to a number of detailed surveys before, during, and after their projects are completed. The surveys include questions regarding the commercial application of proposals, business goals, strategies for commercialization and for protecting intellectual property, dissemination efforts, publication in professional journals and presentations at conferences, participation in user associations, public-relations efforts, R&D status, collaborative efforts, impacts on employment, and attraction of new funding.
Evaluations of the U.S. Air Force research program are conducted by the U.S. Air Force SAB. The first SAB evaluation was conducted in 1991 (R. Selden, U.S. Air Force SAB, personal commun., Jan. 9, 2003). Programs are evaluated for quality and relevance of research, and each directorate is evaluated every 2 years. Typical metrics used to evaluate research programs are university metrics (publications, patents, and peer review) and a grading system that is used to evaluate the various components of the research programs in each directorate on the basis of 10 criteria (see Box C-2). Scores are normalized across the different directorates (Selden 1998).
Many states have developed science and technology performance metrics to evaluate whether and how research programs are encouraging economic development. State governments tend to be more interested in whether research programs are encouraging economic development than in the quality or value of the research itself. Texas, Kansas, and Maine have developed a process of using performance measures to evaluate their re