This appendix is a reprint of the main text of the National Research Council’s Interim Report for the Triennial Review of the National Nanotechnology Initiative, Phase II (The National Academies Press, Washington, D.C., 2012) by the Committee on Triennial Review of the National Nanotechnology Initiative: Phase II.
Nanotechnology has become one of the defining ideas in global research and development (R&D) over the last decade. In 2001, the National Nanotechnology Initiative (NNI) was established as the U.S. government interagency program for coordinating nanotechnology R&D among federal agencies and facilitating communication and collaborative activities in nanoscale science, engineering, and technology throughout the federal government. The NNI defines nanotechnology on its Web site1 as “science, engineering, and technology conducted at the nanoscale, which is about 1 to 100 nanometers.”2 The NNI focuses on four goals aimed at creating “a future in which the ability to understand and control matter at the nanoscale leads to a revolution in technology and industry that benefits society.” The 26 federal agencies that participate in the NNI collaborate to (1) advance world-class nanotechnology research and development, (2) foster the transfer of new technologies into products for commercial and public benefit, (3) develop and sustain educational resources, a skilled workforce, and the supporting infrastructure and tools to advance nanotechnology, and (4) support the responsible development of nanotechnology.
As part of the second triennial review of the NNI, the Committee on Triennial Review of the National Nanotechnology Initiative: Phase II was asked to provide
2 For another definition of nanotechnology, see, for example, National Research Council, A Matter of Size: Triennial Review of the National Nanotechnology Initiative, The National Academies Press, Washington, D.C., 2006.
advice to the Nanoscale Science, Engineering, and Technology (NSET) Subcommittee of the National Science and Technology Council’s Committee on Technology and the National Nanotechnology Coordination Office as follows:
• Task 1—Examine the role of the NNI in maximizing opportunities to transfer selected technologies to the private sector, provide an assessment of how well the NNI is carrying out this role, and suggest new mechanisms to foster transfer of technologies and improvements to NNI operations in this area where warranted.
• Task 2—Assess the suitability of current procedures and criteria for determining progress toward NNI goals, suggest definitions of success and associated metrics, and provide advice on those organizations (government or non-government) that could perform evaluations of progress.
• Task 3—Review NNI’s management and coordination of nanotechnology research across both civilian and military federal agencies.
The present interim report offers the committee’s initial comments on current procedures and criteria for determining progress toward achievement of NNI goals, the proper role of metrics in assessing the NNI, some characteristics of good metrics, and possible metrics and their links to suggested short-term and long-term NNI goals.
This report reflects the committee’s view that measuring something just because it can be measured is not good enough: metrics must be indicators of desired outcomes. There must be a model that accurately relates what is measured to a desired outcome and an equally accurate system to perform the measurement. Having both constitutes a metric. Without both, measurements have little value for program assessment and management.
The committee recognizes the great difficulty in defining robust models and metrics for a field as diffuse as nanotechnology, for agencies as diverse as the 26 NNI participating agencies, and for goals as far-reaching and cross-cutting as the four NNI goals. However, the committee emphasizes that whatever models and metrics are applied must be rigorous and stand up fully to scientific scrutiny. If the data used are inaccurate or if the models linking even accurate data to desired outcomes have not been properly established, evaluation, rational decision-making, and allocation of resources become compromised. In general, computational and data capacities have outrun the accuracy of measurement systems and understanding of the phenomena that relate metrics to desired outcomes. The result may be exciting graphical representations whose meaning remains uncertain. A key part of any solution would be to get scientists in the NNI community to work together to develop models that can be tested to validate current measures. Research on indicators and processes to support metrics would also be highly valuable. In its
final report, the committee will provide recommendations based on the concepts presented in this interim document and will address Tasks 1 and 3 in addition to exploring Task 2 more fully.
The National Nanotechnology Initiative (NNI), a multiagency, U.S. government research and development (R&D) initiative, was established in fiscal year (FY) 2001 to accelerate R&D in the emerging field of nanotechnology:3
The vision of the NNI is a future in which the ability to understand and control matter at the nanoscale leads to a revolution in technology and industry that benefits society. The NNI expedites the discovery, development, and deployment of nanoscale science, engineering, and technology to serve the public good, through a program of coordinated research and development aligned with the missions of the participating agencies.
Starting with eight core agencies in 2001, the NNI now coordinates nanotechnology-related R&D of 26 federal agencies, focusing on four goals (see Box 1.1).
The view of how to achieve the NNI vision has evolved. Starting with the 2004 Strategic Plan, general descriptions of each goal were provided along with selected individual examples. Now the NNI has qualitative, semiquantitative, and quantitative subgoals—as many as five—for each major goal. In addition, the NNI has established five interagency signature initiatives, cross-sector collaborations designed to accelerate innovation in subjects of high national priority through coordination of multiagency resources to meet specific agreed-on scientific and technologic goals; to promote development of joint research solicitations; and to
3 See National Science and Technology Council, National Nanotechnology Initiative Strategic Plan, February 2011, available at http://www.nano.gov/sites/default/files/pub_resource/2011_strategic_plan.pdf. Accessed April 24, 2012.
Goals of the National Nanotechnology Initiative
The National Nanotechnology Initiative focuses on four major goals:
• To advance world-class nanotechnology research and development.
• To foster the transfer of new technologies into products for commercial and public benefit.
• To develop and sustain educational resources, a skilled workforce, and the supporting infrastructure and tools to advance nanotechnology.
• To support the responsible development of nanotechnology.
engage in sponsorship of a wide variety of interagency meetings, workshops, and forums to support knowledge-sharing.
The federal government has given high priority to the alignment of nanotechnology R&D with the missions of the individual agencies. For most agencies, nanotechnology R&D is not an end in itself but rather, in some cases, an enabling technologic means of accomplishing their missions. Each agency determines its budget for nanotechnology R&D as part of its overall mission R&D priorities in coordination with the Office of Management and Budget, the Office of Science and Technology Policy, and Congress. The NNI is planned and coordinated by the Nanoscale Science, Engineering, and Technology (NSET) Subcommittee of the National Science and Technology Council (NSTC) Committee on Technology, through which the agency members present their priorities and establish shared goals, strategies, and activities when their agency priorities align. The 2011 NSET Strategic Plan describes the agencies, their missions, how they view the NNI, and how the NNI fits into their missions. Each NNI participating agency is obliged to carry out its mission and achieve its goals while coordinating and collaborating with other agencies in subjects of mutual interest and mission need.4
To focus interagency collaboration in strategic fields, the NSET Subcommittee has established four cross-agency working groups: Global Issues in Nanotechnology; Nanotechnology Environmental and Health Implications; Nanomanufacturing, Industry Liaison, and Innovation; and Nanotechnology Public Engagement and Communications. The National Nanotechnology Coordination Office (NNCO) provides technical and administrative support to the NSET Subcommittee, serves as the central point of contact for federal NNI R&D activities, and reaches out to
4 Department of Defense Director, Defense Research and Engineering, Defense Nanotechnology Research and Development Program, December 2009. Available at http://www.nano.gov/sites/default/files/pub_resource/dod-report_to_congress_final_1mar10.pdf. Accessed March 3, 2012.
Pursuant to Section 5 of Public Law 108-153, the director of the NNCO requested that the National Research Council conduct the second triennial review of the NNI. The statement of task for the Committee on Triennial Review of the National Nanotechnology Initiative: Phase II is given in Appendix A. The overall objective of the committee’s review is to make recommendations to the NSET Subcommittee and the NNCO that will improve the value of the NNI’s strategy and portfolio for basic research, applied research, and development of applications to provide economic, societal, and national-security benefits to U.S. citizens.
The statement of task reflects the broad attention to and interest in optimizing the federal government’s investments to advance the commercialization, manufacturing capability, national economy, and national security of the United States. For example, the President’s Council of Advisors on Science and Technology (PCAST) 2010 Report to the President and Congress on the Third Assessment of the National Nanotechnology Initiative stated that “the NNCO must develop metrics for program outputs” and “work with the Bureau of Economic Analysis to develop metrics and collect data on the economic impacts of the NNI.”7 The NSET 2011 Strategic Plan established the objective to “develop quantitative measures to assess the performance of the U.S. nanotechnology R&D program relative to that of other major economies, in coordination with broader efforts to develop metrics for innovation.”8 The PCAST 2012 Report to the President and Congress on the Third Assessment of the National Nanotechnology Initiative reiterated its earlier recommendation, calling for the NNCO to “track the development of metrics for quantifying the Federal nanotechnology portfolio and implement them to assess NNI outputs.”9,10
6 See http://www.wtec.org/nano2/Nanotechnology_Research_Directions_to_2020/chapter00-2.pdf. Accessed February 21, 2013.
7 Report to the President and Congress on the Third Assessment of the National Nanotechnology Initiative, President’s Council of Advisors on Science and Technology, March 2010.
8 National Science and Technology Council, National Nanotechnology Initiative Strategic Plan, February 2011, available at http://www.nano.gov/sites/default/files/pub_resource/2011_strategic_plan.pdf. Accessed April 24, 2012.
9 Report to the President and Congress on the Fourth Assessment of the National Nanotechnology Initiative, President’s Council of Advisors on Science and Technology, April 2012.
10 A related study on this subject is the 2012 National Research Council report Improving Measures of Science, Technology, and Innovation: Interim Report (National Academies Press, Washington, D.C., 2012), which examines the current status of science and technology indicators developed and published by the National Science Foundation’s National Center for Science and Engineering Statistics (NCSES) to measure (1) the condition and progress of U.S. science, technology, engineering, and mathematics (STEM) education and workforce development, (2) U.S. innovation and competitiveness in science, technology, and R&D compared with other countries, and (3) whether the NCSES’s
The NNI has now reached a level of achievement and maturity such that its participating agencies are examining the possibility of developing better definitions of success and associated metrics that will guide the agencies individually and the NNI as a whole in expediting “the discovery, development, and deployment of nanoscale science, engineering, and technology to serve the public good”11 to accomplish the four highly integrated NNI goals. This interim report provides the committee’s initial comments related to Task 2: to assess whether the current procedures and metrics are suitable for determining progress toward NNI goals and to suggest alternative definitions of success and their associated metrics. Recommendations related to this task and to Tasks 1 and 3 will be offered in the committee’s final report.
statistical activities are focused properly to produce the information that policy-makers, researchers, and businesses need for decision-making.
11 National Science and Technology Council, National Nanotechnology Initiative Strategic Plan, February 2011, available at http://www.nano.gov/sites/default/files/pub_resource/2011_strategic_plan.pdf. Accessed April 24, 2012.
Observations on the Current Procedures and Criteria for Determining Progress Toward Achievement of National Nanotechnology Initiative Goals
The 26 federal agencies that participate in the National Nanotechnology Initiative (NNI) are listed in Table 2.1; the top 15 in the list have NNI-related programs funded through the federal appropriations process. The eight cross-cutting NNI program component areas (PCAs), which are defined in the 2003 authorizing legislation as major subject areas in which related projects and activities are grouped, are listed in Table 2.2, and the relationships between the PCAs and missions, interests, and needs of the participating NNI agencies are shown in Table 2.3.
In the 2011 NNI Strategic Plan, each agency articulated how nanotechnology had or will have an effect on its achieving its mission and how this maps into the cross-agency PCAs. Examples are provided here in excerpts from the statements made by the Department of Defense (DOD; Box 2.1), the National Institutes of Health (NIH; Box 2.2), and the Department of Labor/Occupational Safety and Health Administration (DOL/OSHA; Box 2.3). Those statements from three representative NNI participating agencies provide a view of what they regard as success for the NNI. For example, DOD seeks “sensors …, communications, and information processing systems needed for qualitative improvements in persistent surveillance,” OSHA seeks to “educate employers on their responsibility to protect workers and educate them on safe practices in handling nanomaterials,”
TABLE 2.1 Agencies Participating in the National Nanotechnology Initiative in 2012
|Federal Agencies with Budgets Dedicated to Nanotechnology Research and Development|
|Agricultural Research Service (U.S. Department of Agriculture, USDA)|
|Consumer Product Safety Commission|
|Department of Defense|
|Department of Energy|
|Department of Homeland Security|
|Department of Transportation (DOT, including the Federal Highway Administration)|
|Environmental Protection Agency|
|Food and Drug Administration (Department of Health and Human Services [DHHS])|
|Forest Service (USDA)|
|National Aeronautics and Space Administration|
|National Institute for Occupational Safety and Health (Centers for Disease Control and Prevention, DHHS)|
|National Institute of Food and Agriculture (USDA)|
|National Institute of Standards and Technology (Department of Commerce [DOC])|
|National Institutes of Health (DHHS)|
|National Science Foundation|
|Other Participating Agencies|
|Bureau of Industry and Security (DOC)|
|Department of Education|
|Department of Justice|
|Department of Labor (including Occupational Safety and Health Administration)|
|Department of State|
|Department of the Treasury|
|Director of National Intelligence|
|Nuclear Regulatory Commission|
|U.S. Geological Survey (Department of the Interior)|
|U.S. International Trade Commission|
|U.S. Patent and Trademark Office (DOC)|
SOURCE: National Science and Technology Council, National Nanotechnology Initiative Strategic Plan, February 2011, available at http://www.nano.gov/sites/default/files/pub_resource/2011_strategic_plan.pdf. Accessed April 24, 2012.
and NIH seeks “new classes of nanotherapeutics and diagnostic biomarkers, tests, and devices.” With respect to collaboration among NNI participating agencies, the 2011 NNI Strategic Plan identified specific subjects for close, targeted interaction, including nanotechnology signature areas, “to foster innovation and accelerate nanotechnology development.”12
The NNI reports progress toward the four NNI goals annually in the NNI supplement to the president’s budget as required by the Nanotechnology Research and Development Act of 2003 (Public Law 108-153). Issued by the Nanoscale Science, Engineering, and Technology Subcommittee of the National Science and
12 National Science and Technology Council, National Nanotechnology Initiative Strategic Plan, February 2011, available at http://www.nano.gov/sites/default/files/pub_resource/2011_strategic_plan.pdf. Accessed April 24, 2012.
TABLE 2.2 National Nanotechnology Initiative Program Component Areas
|Program Component Area||Description|
|Fundamental Nanoscale Phenomena and Processes||Discovery and development of fundamental knowledge pertaining to new phenomena in the physical, biologic, and engineering sciences that occur on the nanoscale. Elucidation of scientific and engineering principles related to nanoscale structures, processes, and mechanisms.|
|Nanomaterials||Research aimed at the discovery of novel nanoscale and nanostructured materials and at a comprehensive understanding of the properties of nanomaterials (ranging across length scales and including interface interactions). Research and development (R&D) leading to the ability to design and synthesize, in a controlled manner, nanostructured materials with targeted properties.|
|Nanoscale Devices and Systems||R&D that applies the principles of nanoscale science and engineering to create novel devices and systems or to improve existing devices and systems. Includes the incorporation of nanoscale or nanostructured materials to achieve improved performance or new functionality. The enabling science and technology must be at the nanoscale, but the systems and devices themselves need not be.|
|Instrumentation Research, Metrology, and Standards for Nanotechnology||R&D pertaining to the tools needed to advance nanotechnology research and commercialization, including next-generation instrumentation for characterization, measurement, synthesis, and design of materials, structures, devices, and systems. Also includes R&D and other activities related to development of standards, including standards for nomenclature, materials characterization and testing, and manufacture.|
|Nanomanufacturing||R&D aimed at enabling scaled-up, reliable, and cost-effective manufacturing of nanoscale materials, structures, devices, and systems. Includes R&D and integration of ultraminiaturized top-down processes and increasingly complex bottom-up or self-assembly processes.|
|Major Research Facilities and Instrumentation Acquisition||Establishment of user facilities, acquisition of major instrumentation, and other activities that develop, support, or enhance the nation’s scientific infrastructure for the conduct of nanoscale science, engineering, and technology R&D. Includes continuing operation of user facilities and networks.|
|Environment, Health, and Safety||Research directed primarily at understanding the environmental, health, and safety effects of nanotechnology development and corresponding risk assessment, risk management, and methods for risk mitigation.|
|Education and Societal Dimensions||Education-related activities, such as development of materials for schools, undergraduate programs, technical training, and public communication, including outreach and engagement. Research directed at identifying and quantifying the broad implications of nanotechnology for society, including social, economic, workforce, educational, ethical, and legal implications.|
SOURCE: National Science and Technology Council, National Nanotechnology Initiative Strategic Plan, February 2011, available at http://www.nano.gov/sites/default/files/pub_resource/2011_strategic_plan.pdf. Accessed April 24, 2012.
TABLE 2.3 Relationships Between Program Component Areas and Missions, Interests, and Needs of Agencies Participating in the National Nanotechnology Initiative
|Fundamental Nanoscale Phenomena and Processes||Nanomaterials||Nanoscale Devices and Systems||Instrumentation Research, Metrology, and Standards||Nanomanufacturing||Major Research Facilities and Instrumentation Acquisition||Environment, Health, and Safety||Education and Societal Dimensions|
NOTE: A check mark denotes a primary relationship and a bullet a secondary relationship. SOURCE: National Science and Technology Council, National Nanotechnology Initiative Strategic Plan, February 2011, available at http://www.nano.gov/sites/default/files/pub_resource/2011_strategic_plan.pdf. Accessed 4/24/2012.
Department of Defense Statement
The following is excerpted from DOD’s statement in the 2011 NNI Strategic Plan.
Department of Defense (DOD) leadership considers nanotechnology to have high and growing potential to contribute to the warfighting capabilities of the nation. Because of the broad and interdisciplinary nature of nanotechnology, DOD leadership views it as an enabling technology area that should receive the highest level of department attention and coordination. The vision and capability construct of Defense Research and Engineering includes nanotechnology as one of four exemplary foundational technologies, along with advanced materials, advanced electronics, and manufacturing technology. DOD Basic Research acknowledges that realizing the potential of nanotechnology is a key research objective. In particular, nanotechnology is an enabling technology for new classes of sensors (such as novel focal plane arrays and chemical/biological threat sensors), communications, and information processing systems needed for qualitative improvements in persistent surveillance. The DOD also invests in nanotechnology for advanced energetic materials, photocatalytic coatings, active microelectronic devices, structural fibers, strength- and toughness-enhancing additives, advanced processing, and a wide array of other promising applications.
SOURCE: National Science and Technology Council, National Nanotechnology Initiative Strategic Plan, February 2011, available at http://www.nano.gov/sites/default/files/pub_2011_strategic_plan.pdf. Accessed 04/24/2012.
National Institutes of Health Statement
The following is excerpted from NIH’s statement in the 2011 NNI Strategic Plan.
The NIH mission is to seek fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to enhance health, lengthen life, and reduce the burdens of illness and disability. Toward this end, NIH leadership realizes that advances in nanoscience and nanotechnology have the potential to make valuable contributions to biology and medicine, which in turn could contribute to a new era in healthcare. The Federal agencies’ R&D investments, for example, have resulted in advanced materials, tools, and nanotechnology-enabled instrumentation that can be used to study and understand biological processes in health and disease. The NIH-supported R&D efforts, in particular, are bringing about new paradigms in the detection, diagnosis, and treatment of common and rare diseases, resulting in new classes of nanotherapeutics and diagnostic biomarkers, tests, and devices.
SOURCE: National Science and Technology Council, National Nanotechnology Initiative Strategic Plan, February 2011, available at http://www.nano.gov/sites/default/files/pub_resource/2011_strategic_plan.pdf. Accessed 04/24/2012.
Department of Labor Statement
The following is excerpted from DOL’s statement in the 2011 NNI Strategic Plan.
The Department of Labor (DOL) Occupational Safety and Health Administration (OSHA) plays an integral role in nanotechnology by protecting the nation’s workforce. Through the NNI interagency efforts, OSHA accomplishes its mission by collaborating and sharing information with other Federal agencies. As part of this effort, OSHA’s goal is to educate employers on their responsibility to protect workers and educate them on safe practices in handling nanomaterials. OSHA is developing guidance and educational materials promoting worker safety and health that will be shared with the public and through the NNI.
SOURCE: National Science and Technology Council, National Nanotechnology Initiative Strategic Plan, February 2011, available at http://www.nano.gov/sites/default/files/pub_resource/2011_strategic_plan.pdf. Accessed 04/24/2012.
Technology Council’s Committee on Technology, the annual supplement includes budget information by agency and by PCA for the prior year (actual spending), the current year (estimated), and the coming year (planned). The NNI also reports the amount of funding that went to nanotechnology-related Small Business Innovation Research (SBIR) awards and Small Business Technology Transfer (STTR) awards to date. (The amount of SBIR and STTR funding that is invested in nanotechnology is not planned, so only prior-year data are available.) The agencies provide examples of specific activities as evidence of progress toward each of the four NNI goals, including coordinated activities with “other agencies, disciplines, industrial sectors, and nations.”13 The annual report released in 2011 (accompanying the president’s FY 2012 budget) included for the first time estimated spending in 2011 and planned spending in 2012 for each of three multiagency signature initiatives (Solar Energy Collection and Conversion, Sustainable Nanomanufacturing, and Nanoelectronics for 2020 and Beyond).
The data on budget and expenditures reported in the annual NNI supplement to the president’s budget and in reports to Congress provide a picture of how resources are being allocated by agency to each of the PCAs. However, progress toward achieving the four NNI goals is reported in largely anecdotal form. Several agencies provide examples of successful projects, some provide numerical data,
13 The National Nanotechnology Initiative, Supplement to the President’s 2013 Budget. Available at http://www.nano.gov/sites/default/files/pub_resource/nni_2013_budget_supplement.pdf. Accessed August 8, 2012.
and some present short summaries without many details. Interagency activities are reported in the same manner. That approach is consistent with how the NNI agencies manage their overall portfolios, how they gather information to report to the president, and what is included in the NNI supplement to the president’s budget.
There is no common method or system across the NNI participating agencies for measuring and tracking progress toward achieving the four NNI goals (see Box 1.1). Broad generalizations about progress are made, but there are few details except for specific examples of successful projects, discoveries, and products related to the agencies’ statements, which are mapped onto the four goals. At the agency level, individual projects are monitored and evaluated with respect to their agreed-on deliverables by using processes and metrics developed by the sponsoring agencies. But such evaluations typically are program-specific, and the deliverables and outcomes are generally reported in forms that cannot be easily aggregated and analyzed. Consider, for example, Goal 1—to advance world-class nanotechnology research and development. The generation of world-class scientific publications, the body of published work associated with an activity, could be considered an indicator of success; metrics would include number of publications, topics, quality of journals, number of citations, and so on. However, there is no comprehensive compilation of publications for NNI-funded R&D for any agency, much less for the whole NNI. The challenge of developing metrics that align with all the NNI goals is the focus of Chapters 3 and 4 of this interim report.
The Role of Metrics
Metrics are necessary for evaluation, rational decision-making, and appropriate allocation of resources. It is useful to distinguish three classes of metrics: for inputs, outputs, and outcomes. Inputs are often measured in dollars spent, in part because such figures are relatively easily determined. Outputs are activity and productivity, whereas outcomes are effects and progress toward overall goals. Outcomes depend heavily on program objectives. Often, inputs are used as a proxy for outputs, but they are generally a poor substitute in that they do not account for the effectiveness or efficiency of a funded activity. A good metric for output should be an accurate measure of whether the desired outcomes of an activity have been achieved—outcomes that represent the value that the activity was intended to generate. In fact, however, many accepted quantitative metrics are used to measure what can be easily measured rather than the value created in the course of the activity.
The relationship between metrics for output and for outcomes of the National Nanotechnology Initiative (NNI) can be illustrated by analogy with manufacturing. In manufacturing, a material or product is measured for three reasons: for quality control, for quality improvement, and to establish that a legal requirement specified in a contract between a supplier and a customer has been met. In the first case, all that is needed is a simple, reliable measure to identify when an acceptable outcome is no longer being produced; measurement yields a result as simple as “acceptable/unacceptable,” and the information that it provides stays local to provide quality control. In the second case, measurement is more quantitative and guides changes to produce better outcomes than previously obtained. In the third case, a supplier agrees to provide to the customer a material that has specific properties as measured
by specific agreed-on, standardized techniques. In each of those cases, there is a well-established model that relates the measurement to the desired outcome, and the measures may be different for the three different functions of metrics.
Applying that to the NNI, many NNI metrics are designed primarily for quality control within the individual agencies on the basis of their individual missions, and many of the possible metrics listed in Chapter 4 of this interim report are in that category. The issue, however, is how to assess the success of the NNI as a whole, as opposed to the success of the individual agencies in fulfilling their missions. Output data gathered by different NNI participating agencies cannot now be usefully compared. The measurement systems are not the same, and the metrics and processes used for quality control are peculiar to each agency, its mission, and its historical way of doing things. Furthermore, researchers and organizations know that they have been funded by a particular agency and are familiar with the agency’s metrics and desired outcomes. The committee learned, in contrast, that many programs and associated researchers do not know that their federally funded research and development (R&D) projects have been included in their funding agencies’ reported NNI program dollars.
METRICS FOR ASSESSING THE NATIONAL NANOTECHNOLOGY INITIATIVE—SOME CONSIDERATIONS
The NNI is being asked to establish metrics for quality improvement, that is, improvement of the NNI and its R&D system for addressing the four NNI goals, and contractual metrics, that is, regarding the effective customer-supplier contracts between, such as taxpayers and the government, Congress and the NNI, principal investigators or companies and the agencies, workers and those who regulate nanotechnology in the workplace, and consumers and agencies that are responsible for food and product safety. For such sets of “customers-suppliers,” there must be a model that relates what is measured—outputs—to the short-term, intermediate-term, and long-term outcomes that the customer is paying for, and there must be an accurate system of quantitative and qualitative metrics that support the model. Without the model, metrics for output probably will lead to an incomplete and inaccurate assessment of whether the outcomes are being met—that is, whether the quality of the NNI program is high, the NNI is increasing its impact, and the NNI is meeting its “contract” with all its “customers.”
Additional characteristics of a good metric are that the information supporting it are reliably and relatively easily obtainable and that, at the very least, the benefits contributed by the metric to evaluation, strategy, and priority-setting justify the cost of obtaining the information. The information generated by the metric also should be able to provide the basis of program decision-making; in other words, it
should be actionable. Many metrics are too general to contribute to the discussion of any specific, important issue.
The quest for good metrics is often framed as a quest for quantitative metrics, which can be measured in an objective way and for which the result is a number or a collection of numbers. However, the emphasis on having objective, numerical metrics often leads to collecting output data that are peripheral to the goals and outcomes of an activity. For example, the number of papers published per year by a researcher is not the only metric of scholarly achievement. Clearly, some consideration of quality and impact of output is also required. Various metrics related to citation may be of partial use in evaluating the quality of a body of publications, but if, for example, the utility of the results presented in publications is the quality or value being sought, citation-count metrics are poor indicators. Furthermore, there is general awareness that the choice of metrics may change the behavior of participants in ways not necessarily conducive to successful outcomes. That is a known and difficult problem that has received considerable attention. Academe’s answer to such problems is to evaluate a person on the basis of a model of academic success that uses a set of subjective, qualitative metrics supported by quantitative data on output and subjective evaluation of the data. That combination of subjective evaluations and quantitative output metrics has evolved to support a model of academic success for faculty at different career stages and performance levels, from assistant to full professor. Dependence on the subjective evaluation of a group of experts chosen for some mix of technical expertise, judgment, and breadth of knowledge of a field is key to this approach. Although the results of the application of qualitative metrics are subjective, such metrics have been demonstrated both to be reasonably reproducible and to encourage desired outcomes successfully; this suggests that the model on which they are based and the methods are reliable. The process has also been developed to ensure that the experts who provide the assessments are in positions of sufficient personal independence from the people being evaluated that they can render objective evaluations.
Notwithstanding those issues, given the investment in and the scope of the NNI, quantitative and qualitative metrics can be applied to assess the impacts of NNI-related activity. Many major federal funders of nanotechnology research are working on the problem of defining a set of quantitative metrics that relate program outputs to desired outcomes in arenas that overlap with the NNI. A prime example is National Institute of Standards and Technology’s (NIST’s) leadership in developing metrics for technology transfer from federal agencies that have research facilities to the commercial marketplace.14 The resulting metrics should be taken into account in the review of NNI activities with qualitative and semiquantitative
14 See http://www.nist.gov/tpo/publications/upload/DOC-FY2011-Annual-Tech-Transfer-DOC.pdf. Accessed August 27, 2012.
assessments by experts. Ideally, such assessments would improve the efficiency, quality, and completeness of the review process. Such a collection of metrics, taken as a whole, may be viewed as an indicator of impact or success and provide guidance for decision-making and for allocation of resources.
Quantitative metrics require various kinds of output and outcome data—such as people trained, jobs created, papers published, awards earned, patents filed, companies started, and products created—measured over time for the agencies or organizations, researchers, and so on. To provide sound input for assessments, those data must be melded in weighted fashion in a manner that respects the missions, nature, and objectives of the responsible agencies or programs. Clearly, uniform models and metrics for all 26 NNI participating agencies are neither practical nor appropriate. Five agencies (the National Science Foundation [NSF], the Department of Defense, the Department of Energy, the National Institutes of Health, and NIST) account for well over 90 percent of the funds and effort expended. The other agencies play different, although still critical, roles in the development of nanotechnology and the NNI. The committee believes that it is important to select output and outcome metrics to minimize the burden on each agency of gathering and reporting data that are not central to its mission or that would require substantial added effort without substantial benefit to the NNI.
The committee recognizes the great difficulty in defining robust models and metrics for a field as diffuse as nanotechnology and for agencies as diverse as the 26 NNI participating agencies. However, it urges that, as difficult as this task may be, whatever models and metrics are applied should be rigorous, that is, should have clearly and publicly defined assumptions, sources, methods, and means to test whether the models and data are accurate. Despite the recognizable value of many of the data provided to and by the NNI agencies and the National Nanotechnology Coordination Office, the origins of the data or assumptions used in collecting or collating the data were not always clear. Furthermore, the committee believes that data arising from “self-identification” or “self-reporting” do not always give an accurate and complete picture of the status of a field. If the data used are inaccurate or if the models or understanding that link even accurate data to desired outcomes have not been well established, evaluation, rational decision-making, and allocation of resources become compromised.
The provenance of data, including the original assumptions and calculations used to develop them, must be clearly established, documented, and maintained. Although source data are not likely to be perfect, the intent should be to make the process of data selection and the results as transparent as possible.
The committee sees promise in many of the aspects of the NSF Star Metrics project15 but also grounds for concern. Directly accessing institutional human-
resources databases to automate data collection on personnel, for example, seems excellent. However, the software algorithms used to parse project summaries to identify emerging fields of research may not be ready for application, given the sample outputs shown to the committee, so implementation of the Star Metrics approach to define fields and current funding levels without independent validation could lead to erroneous conclusions. That observation reflects the state of research that applies machine learning to social-science problems; advances in machine learning and automated inference from large datasets have proceeded rapidly, but validation of the calculated measures has lagged far behind. The lag results from the difficulty of validation, which requires careful sampling of adequate observations for field-work validation, such as interviews, surveys, and historical case studies; lack of collaboration between experts in quantitative data analysis and social-science field research methods; and lack of validated models that relate the output data to the desired outcomes.16
Although software algorithms and data-mining offer promising approaches to data collection, the committee believes that use of a specific set of keywords or field categories, identified by research investigators or program managers, could be improved sufficiently with relatively little effort to be useful for future data collection. However, the committee was surprised to learn that the current software system for project monitoring in NSF, called FastLane—whereby investigators enter data into multiple fields to describe project participants, results, and outcomes, including papers published—apparently could not be used to mine the data supplied by NNI-supported projects.
In general, metrics will be poor if they present misleading information about actual or probable success in accomplishing desired goals, that is, the desired outcomes. There are several characteristics to avoid or minimize in developing metrics. For example, ambiguity in the definition of a metric can lead to combining incoherent data and to analyses of questionable value. Such ambiguity can result from metrics that are too complex. It is better to have simple metrics without too many qualifiers. Another type of problematic metric is one for which optimization of an individual result is easily accomplished at the expense of another important goal, especially if the latter is not captured by a corresponding metric. A great deal of care must be taken to understand the use of specific metrics in different NNI communities and agencies. For example, some communities write more and shorter papers and cite sparsely, whereas others write fewer and longer papers and cite generously. The different practices can produce different distributions of measures of output and impact, and comparisons among fields can become problematic. The effectiveness of a metric may also be compromised by lack of availability or accuracy of the corresponding data, owing, for example, to small samples,
16 G. King, Ensuring the Data-rich Future of the Social Sciences, Science 331:719-721, 2011.
a dearth of accurate sources, estimation errors, and the burden of responding to numerous requests for data. For all those reasons, a model that has a balanced set of metrics should be established.
In summary, the committee finds that strictly quantitative metrics of output are not by themselves dispositive in evaluating the success of the NNI in achieving its goals. Well-crafted qualitative and semiquantitative metrics and their review, supported by quantitative metrics, are more likely to be useful in producing evaluations that measure success and can be applied in setting NNI goals and policy.
A POSSIBLE FRAMEWORK FOR ASSESSING SUCCESS
The goal of this interim report is to consider definitions of success for the NNI (the desired short-term, intermediate-term, and long-term outcomes), metrics, and methods for assessing the NNI’s progress toward its goals.
Establishing the connections between inputs, outputs, and short-term to long-term outcomes is difficult and requires articulation and validation of a model. A possible open framework of a model and system for assessing success in achieving desired outcomes for the individual funding agencies and the NNI as a whole is shown in Figure 3.1. Application programming interfaces and linked databases provide access to input and output data that may be used to trace the connections between inputs, outputs, and some short-term outcomes. Inputs may originate with persons or grants, whereas outputs can include publications and patents or organizations; arrows show explicit connections. The arrows suggest the direction of collaborations or connections between people and organizations, number of times that papers are cited in other publications, and outputs.
Essentially, the framework links NNI research products, including grants, papers, and patents; NNI people; NNI agencies and other corporate, government, and academic institutions; and short-term, intermediate-term, and long-term NNI outcomes. Many of the proposed metrics for assessing output are available to or are under development by various agencies and firms. Google Scholar, for example, has disambiguated and linked the publication and patenting careers of many scientists and inventors (although that effort remains proprietary) and highlights the importance of an open framework. Once in place, such a framework could be used to generate metrics of output at various levels of analysis, including specific awards, principal investigators, institutions, or entire nanotechnology subfields. The resulting metrics for output will require careful validation, as discussed above. Although the framework would require substantial investment in record linkage and disambiguation, it would provide flexibility and allow reuse of investment in different scientific fields and bibliometric databases.
Definitions of Success and Metrics
NATIONAL NANOTECHNOLOGY INITIATIVE GOAL 1: TO ADVANCE WORLD-CLASS NANOTECHNOLOGY RESEARCH AND DEVELOPMENT
Support for nanotechnology research and development (R&D) is the activity most strongly associated with the National Nanotechnology Initiative (NNI) and the one that has received the largest share of funding. The NNI has funded R&D performed by individual investigators, small teams, and large multidisciplinary centers, facilities, and networks of researchers in academe, industry, and government.
Definitions of success that might be applied to NNI Goal 1 include the following:
• A full spectrum of R&D, including fundamental research, “use-inspired” basic research, application-driven applied research, and technology development is being supported.
• The NNI supports research that crosses boundaries—research that is multidisciplinary, multi-institutional, multinational, multiagency, and multisectoral (government-university-industry).
• The performance of the U.S. NNI is comparable with or better than that of the best in the rest of the world.
• An appropriate scientific and technical workforce is being trained and educated, and it contributes effectively to the U.S. economy. (See Goal 3.)
• The frontiers of knowledge are being substantially advanced in a way that is commensurate with the scale of funding.
• NNI-supported research is world-class.
• NNI-supported research is leading to valuable new technology. (See Goal 2.)
• Industrial, sector-specific nanotechnology knowledge is used to inform application-driven research investment decisions.
• NNI dollars are spent wisely to advance world-class R&D efficiently and effectively.
• Cohesive and substantial facilities and networks that are of broad relevance to the nanotechnology community are being built, and these facilities foster collaboration. (See Goal 3.)
Possible metrics of progress toward success as defined above for Goal 1 are outlined below.
• Spectrum of R&D assessment funded or supported, on the basis of expenditures categorized according to the following:
— “Basic research,” “applied research,” or “technology development” based on definitions of the Office of Management and Budget or definitions similar to those used by the Department of Defense (DOD) (6.1, 6.2, and so on).17 (Understanding the distribution among these categories over time can help to ensure a balanced portfolio and to track the maturation of nanotechnology from a primarily basic-research endeavor to one that includes substantial application and development investments.)
— Distribution of funds by size of grant, to assess coverage of large and small projects.
— Nature of research performers (academic, government, small, midsize, or large company, nonprofit). Collaboration among sectors should be noted because such collaborations are important for knowledge diffusion and translation to applications (especially if industry is involved).
• Number of publications based on NNI-funded R&D, with analysis of authorship to assess the share that is multidisciplinary, multidepartmental, multiuniversity, multinational, and multisectoral (for example, academe-industry or academe-government).
• Number of publications in the array of disciplines and sectors related to nanotechnology.
• Number of citations to NNI-funded publications by other publications, with additional analysis to assess share of citations that are by authors in industry, another discipline, outside the United States, and other characteristics.
17 See http://www.rand.org/content/dam/rand/pubs/monograph_reports/MR1194/MR1194.appb.pdf. Accessed February 22, 2013.
• Number of citations to NNI-funded publications by patents, with additional analysis of the patent subject categories—or classifications—in which the citations are made.
• Number of patents and patent applications based on NNI-funded research.
• Keynote and invited presentations on NNI-funded R&D at conferences throughout the various disciplines and sectors affected by nanotechnology. Such presentations are generally made by highly regarded researchers and so are a measure of research quality and a measure of diffusion of NNI research.
• Awards and prizes that recognize NNI-supported research that has had a substantial impact, such as awards by selected professional societies and agencies.
• Numbers of scientists, engineers, and technicians trained in nanotechnology, with additional analysis to show what jobs they have moved into. (See also Goal 3.)
As noted above, however, metrics like those are not an end in themselves. The most relevant numerical metrics must serve as the basis of a rational model of the evolution of the NNI R&D system that can be used to assess progress toward the NNI goals. Ideally, quantitative and qualitative metrics is combined with expert assessment whenever possible.
NATIONAL NANOTECHNOLOGY INITIATIVE GOAL 2: TO FOSTER THE TRANSFER OF NEW TECHNOLOGIES INTO PRODUCTS FOR COMMERCIAL AND PUBLIC BENEFIT
A definition of success that might be applied to NNI Goal 2 is the development in the United States of vibrant, competitive, and sustainable industry sectors that use nanotechnology to enable the creation of new products; skilled, high-paying jobs; and economic growth. The committee is keenly aware of the different time frames associated with the transition from discovery to products that are related to the missions of the NNI participating agencies. Some agencies (or offices in agencies) will pursue technologies that are closer to market to address mission-driven needs and goals, whereas others will develop knowledge that may well be many years from or not specific to commercialization. The NNI, like many federal R&D programs, funds primarily activities that are focused on discovery as opposed to commercialization. Commercialization requires private-sector investments over which the NNI has weak influence, so the NNI tends to focus on startups as opposed to large or multinational corporations. One example of an exception to that is the Nanoelectronics Research Initiative, a jointly funded venture between the National Science Foundation (NSF), the National Institute of Standards and
Technology (NIST), and the Semiconductor Research Corporation. Models and metrics for success require an understanding of the pathways and timelines for translation of discovery to commercial products.
Defining commercial benefits within the narrow confines of the U.S. economy is also challenging, given the highly interconnected global economy into which new nanotechnologies are launched. For example, it is extremely difficult to prepare sound economic-impact statements for a new technology that may be invented in the United States but then sold to a company that is headquartered elsewhere. The company may choose to manufacture the nanotechnology-enabled products in a third country but sell them in the United States, possibly yielding improvements in domestic productivity or quality of life, an increase in commercial activity, and financial benefits to U.S.-based shareholders in the company.
Because of such complexities, which are difficult to tease apart, the committee believes that the most robust indicator of commercial benefit to the United States may be the growth of U.S.-based jobs related to nanotechnology. Once that growth is defined and enumerated, pre-existing estimates of the economic good associated with each additional skilled technology worker could be used to extrapolate from the number of jobs to a direct impact on the U.S. economy.
Possible metrics of progress toward success as defined above in achieving NNI Goal 2 are listed below.
• Growth of nanotechnology-related jobs.
• Number of NNI-funded students who are hired for nanotechnology-related jobs.
• Number of published patents and applications (as reported by the U.S. Patent and Trademark Office) and patent licensing categorized according to
— Inventor affiliation (academe, industry, government, individual).
— Subject or sector (electronics, chemicals and materials, and so on).
— Inventor’s country of origin.
• Number of Small Business Innovation Research (SBIR) awards related to nanotechnology, categorized by field of interest or topic.
• Number of nanotechnology-related companies partnering in specific ways with NNI-funded user centers, possibly weighted by funding levels.
• Number and economic health of companies started by NNI-funded SBIR and Small Business Technology Transfer (STTR) recipients.
• Nanotechnology-enabled products known to have been derived at least in part from NNI-funded activities.
Progress in fostering the transfer of technologies into products for commercial and public benefit is difficult to define, assess, and quantify throughout the NNI given the complexity of interactions. The translation of NNI research into products
will require different metrics for different agencies because the products will differ considerably in their type and path to fruition. Translational entities and programs set up by such agencies as DOD, the Department of Energy (DOE), NIST, and NSF may be dedicated to nano-enabled products or have goals that include nanoenabled products. Products vary considerably; for example, the products of NSF-funded university research are typically graduates, publications, and, to a smaller extent, intellectual property, all of which contribute to the development of the nanotechnology workforce and to the body of knowledge. DOD research is generally aimed at developing technology that can be deployed for the national defense. Many companies are interested in products and services for public sale. Standards developed by various standards-development organizations with the participation of NIST and other federal agencies are a public good that supports industry while reducing technical barriers that favor a particular company’s or country’s agenda.
The pathways by which research results are translated into practical applications and commercial products are complex and numerous. Moreover, the time from research to product is typically measured in years or even decades. The NNI has existed for 10 years; nanotechnology-based products are emerging, and many more useful discoveries are in the innovation pipeline. At the agency or industry level, mechanisms exist for technology transfer and commercialization, and different metrics may be required to capture their effectiveness. Moreover, commercialization depends on various innovation activities, and hence various metrics, in the NNI: knowledge generation and dissemination, technology transfer, commercialization, and workforce creation in which NNI agencies and program managers and members of the international nanotechnology R&D community are prime actors. Metrics may be based, for example, on knowledge (publications, intellectual property, and citations), workforce training (graduates, employees, and meetings attended), private-sector engagement (patent licensing data, SBIR or STTR grant data and later venture funding acquisition, cooperative R&D agreements, and public-private partnerships), or revenue.
Desired outputs depend strongly on the agency involved; 26 agencies have widely different levels of engagement in the NNI as measured by funding for the research or staff involved. Outputs may even vary within a single agency. In DOE, for example, NNI-related output includes user centers, Advanced Research Projects Agency-Energy grants and contracts, SBIR funding, and the establishment of the Energy Frontier Research Centers program. In addition, outputs represent a broad range of technology readiness levels, and this has implications for the amount of funding, time, and effort required to convert a discovery or an invention into a useful product.
Encouraging inventors to take risks to commercialize their ideas is as much a cultural issue as it is a financial or a technical issue. Commercialization can be stifled in an environment in which risk-taking is not encouraged, mentors are
not available, or licensing is difficult; some regions and institutions are good at off entrepreneurial activities, and others are not. Those cultural issues are common to universities, government laboratories, and other research institutions and can create a bottleneck in the innovation pipeline. Although that is not a nanotechnology-specific problem, addressing it is important for removing barriers to commercialization of results, given the substantial investment in the NNI.
Inventors and organizations may not be aware of the potential commercial value of technology if there is not an environment that encourages startups or spinoffs, and they may need a mechanism like a “preseed” workshop or NSF I-Corps18 to foster commercialization concepts. Federal and local agencies have recognized that—the NSF I-Corps is an example of what can be done at the federal level to encourage and stimulate growth. It works to connect NSF-funded scientific research to the technologic, entrepreneurial, and business communities. The I-Corps curriculum is built on an accelerated version of Stanford University’s Lean LaunchPad course and additional elements designed for I-Corps grantees. All I-Corps team members attend a kickoff workshop at Stanford University, the Georgia Institute of Technology, or the University of Michigan and then join a series of Web-based lectures and present their business pitches at a meeting of I-Corps grantees. Awards are for $50,000 with a duration of 6 months.
Many other excellent programs of this type may be available throughout the United States, but there is no current way to know how many and where they are. A measure of success for the NNI might be to expedite and facilitate connections for inventors in the nanotechnology-products realm to help them to identify agencies—federal, state, regional, and local—that can support them. The committee will examine such issues in its final report.
NATIONAL NANOTECHNOLOGY INITIATIVE GOAL 3: TO DEVELOP AND SUSTAIN EDUCATIONAL RESOURCES, A SKILLED WORKFORCE, AND THE SUPPORTING INFRASTRUCTURE AND TOOLS TO ADVANCE NANOTECHNOLOGY
The 2011 NNI Strategic Plan notes that the development and sustainment of the infrastructural elements addressed by NNI Goal 3 are essential for delivering commercial and public benefit from NNI efforts. The Strategic Plan supplements Goal 3 with three objectives that are paraphrased here as workforce development, informal education activities, and physical infrastructure development.
Definitions of success that might be applied to NNI Goal 3 include the following:
• The supply matches the demand for U.S.-based skilled nanotechnology workers.19
• Public understanding of and interest in nanotechnology and how it may affect our lives are expanded.
• The amount and the type of infrastructure for nanotechnology advancement are appropriate, given the funding levels.
• Users’ technical needs are met through NNI user facilities.
• Rates of use of NNI infrastructure are high.
Possible metrics of progress as defined above in achieving NNI Goal 3 are listed below.
• Evidence that U.S.-based skilled nanotechnology workers trained through the NNI are fully employed.
• Evidence that there is not unmet demand for skilled nanotechnology workers.
• Numbers of people beyond the NNI research community reached by specific agency-driven outreach activities, such as teacher-education activities and K-12 student activities.
• Mass-media stories about nanotechnology activities in or related to NNI participating agencies.
• Use of current infrastructure, according to numbers and types of users, and the outcomes of use of the infrastructure.
• Satisfaction among participants in user facilities, as established through surveys.
• Responsiveness to unmet needs for infrastructure signaled by unfulfilled requests for access to infrastructure.
The committee is impressed by the number and nature of programs targeting the training of a skilled nanotechnology workforce in the NNI environment. It is in the nation’s interest that the supply of and demand for skilled workers be in balance. It is therefore desirable to collect reliable data on the supply of and demand for workers who have critical skills. Even the number of students who are receiving formal, career-oriented, “nanotechnology” education at various levels funded by NNI agencies is difficult to assess with the current system for collecting data from the agencies that participate in the NNI; only some agencies appear to collect such data, and the National Nanotechnology Coordination Office does not aggregate the data that are available as far as the committee can tell. The committee is considering ways in which data on the supply of workers at all levels of training and education
19 A “nanotechnology worker” is, for example, a scientist or an engineer (such as a materials scientist, chemist, or physicist) who is trained to work on processes in the 1- to 100-nm range.
might be aggregated and compared with indicators of the workforce demand for skilled nanotechnology workers as a function of time.
At a minimum, the NNI-funded ecosystem should be graduating students at a rate sufficient to drive the nanotechnology innovation and commercialization process. Achieving that result, however, will require as a first step the collection and analysis of data. It may, however, be useful to collect and analyze the supply-side dataset. For NNI participating agencies, it may be possible to report where students work immediately after graduation. NNI-trained students moving to employment with U.S. firms, agencies or with institutions involved in nanotechnology could perhaps be fairly viewed as expanding the skilled nanotechnology workforce, whether or not job listings specify nanotechnology skills.
It is difficult to estimate the size of the current nanotechnology workforce, but the related issue of workforce growth in this segment, as estimated from periodic review of U.S. job listings, might provide a useful metric. The committee notes with interest the data on nanotechnology job openings collated by Freeman and Shukla for 2008 directly from the on-line job board SimplyHired.com.20 The data are broken down into 18 categories, some of which are nanotechnology-specific (for example, scientist and engineer) and some of which might be considered support roles (information technology, human resources, and administration). Taken together, however, the data indicate the health of the U.S. nanotechnology economy. If tracked over a longer period, they might be considered a proxy indicator of the growth of the U.S. nanotechnology economy through the demand for a skilled nanotechnology workforce. The committee notes that many of the job listings represent workforce churn—skilled people changing jobs—rather than new positions, so it is the time-based growth in the number of listings that is of primary interest for NNI metrics, given the assumption that the churn rate might be taken as a somewhat constant fraction, other factors being equal.
The number of people receiving “nanotechnology” education at various levels through outreach and informal educational activities enabled by the NNI and the effectiveness of such activities will probably also be important to quantify. It will be difficult to measure efforts to expand public understanding of nanotechnology and all that it entails or to measure the effectiveness of such efforts. A possible metric is an estimate of the number of people reached by specific agency-driven outreach activities.
The NNI has created a substantial infrastructure that includes everything from laboratory equipment that is used by a single principal investigator to major facilities that are open to qualified researchers. The latter category includes the DOE nanoscale science research centers, the NIST Center for Nanoscale Science and
20 R. Freeman and K. Shukla, Jobs in Nanotechnology—Creating a Measure of Job Growth, Science and Engineering Workforce Project Digest, National Bureau of Economic Research, June 2008.
Technology, the National Institutes of Health (NIH)-Food and Drug Administration (FDA)-NIST Nanotechnology Characterization Laboratory, and NSF centers and networks, including the National Nanotechnology Infrastructure Network and the Network for Computational Nanotechnology.21
The committee applauds the objective stated in the 2011 NNI Strategic Plan of taking an inventory of current infrastructure and estimating infrastructure needs out to 2020. The related issue of accessibility of that infrastructure should also be addressed. Metrics of progress toward that objective should track how useful the current infrastructure is (for example, on the basis of numbers and types of users, rates of use of key tools, and outcomes of using the infrastructure) and whether there are unmet infrastructure needs.
The committee is also interested in metrics that indicate the relative success of different models for operating the existing nanotechnology facilities in supporting innovation, such as papers written by academic and industry partners and related patent activity. Such metrics might reveal which operating models are most effective and thus provide direction to the management teams in new and existing facilities that are seeking to maximize impact. Some such data are given in the 2011 report Assessment of Fifteen Nanotechnology Science and Engineering Centers’ (NSECs) Outcomes and Impacts: Their Contribution to NNI Objectives and Goals.22
NATIONAL NANOTECHNOLOGY INITIATIVE GOAL 4: TO SUPPORT THE RESPONSIBLE DEVELOPMENT OF NANOTECHNOLOGY
NNI Goal 4 attempts “to assure that nanotechnology-enabled products minimize adverse impacts and maximize benefits to humans and the environment.” The NNI role in supporting responsible development includes investing in research on potential risks to health or the environment from nanomaterials and on societal aspects of the development of nanotechnology applications. Ensuring responsible development also entails communicating relevant information with various stakeholders, including business, international governance and other organizations, educators, and the public. It is notable that success in responsible nanotechnology development is considered necessary for the achievement of NNI Goals 1-3. Of the eight NNI program component areas, two in particular reflect the goals of responsible development of nanotechnology: Environmental Health and Safety (EHS), and Education and Societal Dimensions.
The 2011 NNI Environmental, Health, and Safety Research Strategy23 lays out the breadth and complexity of NNI Goal 4 and supplements it with a number of important, and in many cases concrete, objectives. In 2012, the funding for EHS is estimated to increase by about 20 percent over 2011 levels. The increase is in keeping with the perception that EHS will be critical for success in leveraging nanotechnology for societal benefit by identifying and addressing potential hazards of nanomaterials at an early stage. The primary agencies, by dollar value, that are supporting the EHS program component area are NSF, NIH, the Environmental Protection Agency, and the National Institute for Occupational Safety and Health, and FDA is playing an increasing role as new nanotechnology products come to market. Although the Consumer Product Safety Commission has been a member of the NNI since 2004, it contributed to the NNI budget for the first time in 2011; this shows the increasing importance of Goal 4 as nanotechnology matures.
Because of the complexity of NNI Goal 4, related definitions of success are particularly challenging to distill but may include the following:
• Development, updating, and implementation of a coordinated program of EHS research leads to development of tools and methods for risk characterization and risk assessment in general—including both hazards and the likelihood of exposure—and supports expanding understanding of potential risks posed by broad classes of nanomaterials.
• Results of EHS research worldwide are public and easily available to researchers and users of nanomaterials.
• Businesses of all sizes are aware of potential risks posed by nanomaterials and know where to obtain current information about their properties and best practices for handling them.
• To enable continued innovation, regulatory agencies have sufficient information to assess the risks posed by new nanomaterials.
• The NNI supports research to assess the societal effects of nanotechnology in parallel with technology development.
• K-12 students are exposed to nanotechnology as part of their education and are aware of the potential applications and opportunities available to those who go into STEM (science, technology, engineering, and mathematics) disciplines.
• The general public has access to information about nanotechnology and a growing percentage is familiar with the fundamental concepts.
23 See http://www.nano.gov/sites/default/files/pub_resource/nni_2011_ehs_research_strategy.pdf. Accessed September 27, 2012.
• The NNI includes R&D aimed at applying nanotechnology to solve societal challenges, such as affordable access to clean water, safe food, and medical care.
Possible metrics of progress toward success as defined above in achieving NNI Goal 4 are listed below.
• EHS collaborations and projects or centers funded.
• Number of NNI EHS research results that are made easily accessible, for example, through an NNI-managed clearinghouse or in cooperation with international organizations.
• Guidance documents developed and made available to the public.
• Number of faculty and students supported for research in nanotechnology-related endeavors.
• Number of K-12 students and educators engaged by NNI-funded researchers, including DOE laboratory outreach and NSF-funded researchers, and the effects of such engagement.
• Evidence of public awareness and attitude regarding nanotechnology based on data on NNI-funded research.
• Availability of on-line information and news items related to nanotechnology.
• Evidence that NNI agencies are engaged in international forums discussing and developing standards, norms, and strategies for responsible development of nanotechnology.
• Number of NNI participating agency representatives at various international forums.
• Compilation of commercialized or commercializable technologies.
• Number of companies offering EHS, nanotoxicity, or nanotechnology safety services.
• Evolution of outcomes and impact of sustained funding in the EHS and societal dimensions of the NNI.
Progress toward Goal 4 requires collection of data and development of methods to assess potential risks associated with engineered nanomaterials. Integral to that effort is the design of methods and protocols for assessing properties of nanomaterials and their biologic effects on the environment and on human health and the creation of guidance documents, standards, or other regulatory approaches. The amount of information that is needed to make informed decisions is large (and expensive to collect and catalog). The committee applauds the NNI for its renewed commitment to addressing these hard problems and plans in its final report to suggest metrics for gauging progress or success without imposing undue reporting burdens on the participating agencies.
THE PATH FORWARD TO IMPROVED METRICS
The committee believes in the value of metrics—why we have them, what we hope to accomplish by using them, and how we can tailor them to yield the information desired—but will not recommend measuring something simply because it can be measured. Metrics should make clear what the desired outcomes are. That is, there must be a model that relates what is measured to the desired outcome and an accurate system for doing the measuring. Having both constitutes having a metric. Without both, measurements will have little value for program management.
The committee recognizes the difficulty of defining robust models and metrics for a field as diffuse as nanotechnology, for agencies as diverse as the 26 NNI participating agencies, and for goals as far-reaching as the four NNI goals. However, it emphasizes that any models and metrics applied must be rigorous and able to stand up fully to scientific scrutiny. If the data used are inaccurate or if the models linking data to desired outcomes have not been properly established, evaluation, rational decision-making, and allocation of resources become compromised. For example, the definitions by various stakeholders of what counts as nanotechnology are not consistent and make comparing or combining current analyses difficult or impossible.
The committee observes that data gathered by different agencies cannot now be usefully compared. The measurement systems are not the same. The agencies use different metrics for their R&D programs that are based on a given agency, its mission, and its historical way of doing things. The NNI is being asked to establish definitions of success and associated metrics for fulfilling the overarching NNI goals while meeting the needs and supporting the missions of the NNI participating agencies. To achieve those objectives, there must be both a model (or a set of models) that relates what is measured to the planned NNI outcomes and an accurate measurement system that operates throughout the NNI agencies. With respect to NNI R&D, some outcomes can be measured now; others may be measurable soon with the use of new data-collection and data-mining capabilities. In sum, what is needed to assess the NNI’s progress and success are accurate measurement systems and valid models. In general, computational and data capacities have outrun the accuracy of measurement systems and understanding of the phenomena that relate metrics to desired outcomes. The result may be exciting graphical representations whose meaning remains uncertain. A key part of the solution is to get scientists together and to work with the NNI community to develop models that can be tested to validate the measures on the ground. In other words, the NNI could benefit from investing in research on indicators and processes to support the development and effective use of metrics.
The issue of metrics is not peculiar to the NNI. Other federal research programs and the international R&D community also are grappling with the issue of how to
measure impact and return on investment. The committee views the present study as an opportunity to stimulate additional discussion on the question of metrics. It believes that metrics and models that relate metrics to outcomes of R&D can and should be developed for the NNI and other government programs. This interim report presents an overview of considerations related to the characteristics of good metrics. The committee’s final report will provide specific recommendations on the topic that are based on the concepts presented here.