found in Gault (2010), who outlines four ways that indicators are used for policy purposes: monitoring, benchmarking, evaluating, and “foresighting.”³

At the panel’s workshop, several presenters described attributes of indicators that NCSES should keep in mind as it develops new STI indicators. One important desirable attribute that was emphasized is a low sensitivity to manipulation. In addition, STI indicators are like baseball statistics—it is unlikely that one single statistic tells the whole story. Instead, users will need to rely on a collection or suite of indicators. Mindfully, during the workshop, Hugo Hollanders, of UNU-MERIT,⁴ stated that there is both political and media appeal of composite indices.⁵ Other ideal characteristics of indicators that workshop participants mentioned included scientifically derived/evidence based; comparable across regions; powerful for communication; affordable; accessible; scalable; sustainable; and policy and analytically relevant. STI indicators need to be policy neutral, even though the particular ones selected may reflect the preferences of the stakeholders who request them.

Although the production of indicators across many fields has an established history, there are at least three major cautions regarding their use that are important to note.

• First, indicators can send mixed signals, which require expert judgment for interpretation. For example, increased innovation—which is key to advancing living standards—is often considered to enhance job creation. Policy makers discuss spurring innovation as a job-creation tactic. However, innovation can lead to fewer jobs if the process or managerial expertise increases efficiency. Short-term displacement of workers in one industry or sector can be counterbalanced in the longer term by development of new products, services, and even sectors and by increased market demand if process efficiencies drive down prices (see Pianta, 2005; Van Rennen, 1997). One way to be cautious about mixed signals is to develop STI indicators that support analysis of time scales, sectors, and geographic locations.
• Second, a given metric, once it becomes widely used, changes the behavior of the people and practices it attempts to measure. The worst thing a metric can do is not just to deliver a bad (i.e., misleading) answer, but to incentivize bad practice (see, e.g., West and Bergstrom, 2010). It is important that indicators avoid sending distorted signals to users.
• Not everything that counts can be counted, and not everything that can be counted counts (an idea attributed to Albert Einstein). It seems clear that some outcome measures that reflect the importance of research and development (R&D) and innovation to society are illusive. For example, social well-being is difficult to measure, yet one of the key interests of policy makers is the return on investment of public funding for science and technology for the good of society.

³For example, at the panel’s workshop Changlin Gao reported that China is targeting its STI indicators’ program on the four broad measures: (1) monitoring (international innovation system, linkages within and between national innovation systems; regional innovation systems and industrial clusters; firms; innovation; the implementation of national S&T projects; the selected quantitative indicators in the S&T development goals); (2) evaluating (performance of public investment on S&T; performance of government research institutes and national labs; national S&T programs; specialization of S&T fields; advantages versus disadvantages; new emerging industries, such as information technology, biotechnology industry, energy, health, knowledge based services, and etc.); (3) benchmarking (international benchmarking; interprovincial benchmarking); and (4) forecasting (the latest data not available in gathered statistics).

⁴UNU-MERIT—the U.N. University Maastricht Economic and Social Research Institute on Innovation and Technology—is a research and training center of the United Nations University and works in close collaboration with the University of Maastricht.

⁵To clarify, the panel is not advocating that NCSES develop a “headline indicator.” A suite of key STI indicators should be more informative for users of the statistics.