In its interim report, the committee examined the role of metrics in managing such programs as the NNI.3 It is most important that measurements be made only if actions will be taken as a result. With a materials-manufacturing analogy, the information in Box 4.1 sheds light on the general relationship between making measurements and taking action as a result.
The following excerpts from the interim report highlight some additional thoughts that guided the committee through the process of writing this chapter and defining its recommendations. For the full text of the committee’s interim report, see Appendix E.
This report reflects the committee’s view that measuring something just because it can be measured is not good enough: metrics must be indicators of desired outcomes. There must be a model that accurately relates what is measured to a desired outcome and an equally accurate system to perform the measurement. Having both constitutes a metric. Without both, measurements have little value for program assessment and management. (p. 141)
… progress toward achieving the four NNI goals is reported in largely anecdotal form. Several agencies provide examples of successful projects, some provide numerical data, and some present short summaries without many details. Interagency activities are reported in the same manner. That approach is consistent with how the NNI agencies manage their overall portfolios, how they gather information to report to the president, and what is included in the NNI supplement to the president’s budget. (pp. 152 and 153)
A good metric for output should be an accurate measure of whether the desired outcomes of an activity have been achieved—outcomes that represent the value that the activity was intended to generate. In fact, however, many accepted quantitative metrics are used to measure what can be easily measured, rather than the value created in the course of the activity. (p. 154)
Additional characteristics of a good metric are that the information supporting it are reliably and relatively easily obtainable and that, at the very least, the benefits contributed by the metric to evaluation, strategy, and priority setting justify the cost of obtaining the information. (p. 155)
The definitions of success and associated metrics that have been applied to NNI-funded programs are set by the agencies, and are, therefore, predominantly agency-mission-based, with nanotechnology being secondary. More is needed for assessing the success of the NNI as a whole beyond the success of the individual agencies in fulfilling their missions. As noted in a 2012 Government Accountability
3 National Research Council, Interim Report for the Triennial Review of the National Nanotechnology Initiative, Phase II, The National Academies Press, Washington, D.C., 2012 (reprinted in Appendix E).