Click for next page ( 17


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 16
4 Definitions of Success and Metrics NATIONAL NANOTECHNOLOGY INITIATIVE GOAL 1: TO ADVANCE WORLD-CLASS NANOTECHNOLOGY RESEARCH AND DEVELOPMENT Support for nanotechnology research and development (R&D) is the activity most strongly associated with the National Nanotechnology Initiative (NNI) and the one that has received the largest share of funding. The NNI has funded R&D performed by individual investigators, small teams, and large multidisciplinary centers, facilities, and networks of researchers in academe, industry, and government. Definitions of success that might be applied to NNI Goal 1 include the following: A full spectrum of R&D, including fundamental research, “use-inspired” basic research, application-driven applied research, and technology development is being supported. The NNI supports research that crosses boundaries—research that is multidisciplinary, multi- institutional, multinational, multiagency, and multisectoral (government-university-industry). The performance of the U.S. NNI is comparable with or better than that of the best in the rest of the world. An appropriate scientific and technical workforce is being trained and educated, and it contributes effectively to the U.S. economy. (See Goal 3.) The frontiers of knowledge are being substantially advanced in a way that is commensurate with the scale of funding. NNI-supported research is world-class. NNI-supported research is leading to valuable new technology. (See Goal 2.) Industrial, sector-specific nanotechnology knowledge is used to inform application-driven research investment decisions. NNI dollars are spent wisely to advance world-class R&D efficiently and effectively. Cohesive and substantial facilities and networks that are of broad relevance to the nanotechnology community are being built, and these facilities foster collaboration. (See Goal 3.) Possible metrics of progress toward success as defined above for Goal 1 are outlined below. Spectrum of R&D assessment funded or supported, on the basis of expenditures categorized according to the following: —“Basic research,” “applied research,” or “technology development” based on definitions of the Office of Management and Budget or definitions similar to those used by the Department of Defense (DOD) (6.1, 6.2, and so on).1 (Understanding the distribution among these categories over time can help to ensure a balanced portfolio and to track the maturation of nanotechnology 1 See http://www.rand.org/content/dam/rand/pubs/monograph_reports/MR1194/MR1194.appb.pdf. Accessed February 22, 2013. 16

OCR for page 16
from a primarily basic-research endeavor to one that includes substantial application and development investments.) —Distribution of funds by size of grant, to assess coverage of large and small projects. —Nature of research performers (academic, government, small, midsize, or large company, nonprofit). Collaboration among sectors should be noted because such collaborations are important for knowledge diffusion and translation to applications (especially if industry is involved). Number of publications based on NNI-funded R&D, with analysis of authorship to assess the share that is multidisciplinary, multidepartmental, multiuniversity, multinational, and multisectoral (for example, academe-industry or academe-government). Number of publications in the array of disciplines and sectors related to nanotechnology. Number of citations to NNI-funded publications by other publications, with additional analysis to assess share of citations that are by authors in industry, another discipline, outside the United States, and other characteristics. Number of citations to NNI-funded publications by patents, with additional analysis of the patent subject categories—or classifications—in which the citations are made. Number of patents and patent applications based on NNI-funded research. Keynote and invited presentations on NNI-funded R&D at conferences throughout the various disciplines and sectors affected by nanotechnology. Such presentations are generally made by highly regarded researchers and so are a measure of research quality and a measure of diffusion of NNI research. Awards and prizes that recognize NNI-supported research that has had a substantial impact, such as awards by selected professional societies and agencies. Numbers of scientists, engineers, and technicians trained in nanotechnology, with additional analysis to show what jobs they have moved into. (See also Goal 3.) As noted above, however, metrics like those are not an end in themselves. The most relevant numerical metrics must serve as the basis of a rational model of the evolution of the NNI R&D system that can be used to assess progress toward the NNI goals. Ideally, quantitative and qualitative metrics is combined with expert assessment whenever possible. NATIONAL NANOTECHNOLOGY INITIATIVE GOAL 2: TO FOSTER THE TRANSFER OF NEW TECHNOLOGIES INTO PRODUCTS FOR COMMERCIAL AND PUBLIC BENEFIT A definition of success that might be applied to NNI Goal 2 is the development in the United States of vibrant, competitive, and sustainable industry sectors that use nanotechnology to enable the creation of new products; skilled, high-paying jobs; and economic growth. The committee is keenly aware of the different time frames associated with the transition from discovery to products that are related to the missions of the NNI participating agencies. Some agencies (or offices in agencies) will pursue technologies that are closer to market to address mission-driven needs and goals, whereas others will develop knowledge that may well be many years from or not specific to commercialization. The NNI, like many federal R&D programs, funds primarily activities that are focused on discovery as opposed to commercialization. Commercialization requires private-sector investments over which the NNI has weak influence, so the NNI tends to focus on startups as opposed to large or multinational corporations. One example of an exception to that is the Nanoelectronics Research Initiative, a jointly funded venture between the National Science Foundation (NSF), the National Institute of Standards and Technology (NIST), and the Semiconductor Research Corporation. Models and metrics for success require an understanding of the pathways and timelines for translation of discovery to commercial products. 17

OCR for page 16
Defining commercial benefits within the narrow confines of the U.S. economy is also challenging, given the highly interconnected global economy into which new nanotechnologies are launched. For example, it is extremely difficult to prepare sound economic-impact statements for a new technology that may be invented in the United States but then sold to a company that is headquartered elsewhere. The company may choose to manufacture the nanotechnology-enabled products in a third country but sell them in the United States, possibly yielding improvements in domestic productivity or quality of life, an increase in commercial activity, and financial benefits to U.S.-based shareholders in the company. Because of such complexities, which are difficult to tease apart, the committee believes that the most robust indicator of commercial benefit to the United States may be the growth of U.S.-based jobs related to nanotechnology. Once that growth is defined and enumerated, pre-existing estimates of the economic good associated with each additional skilled technology worker could be used to extrapolate from the number of jobs to a direct impact on the U.S. economy. Possible metrics of progress toward success as defined above in achieving NNI Goal 2 are listed below. Growth of nanotechnology-related jobs. Number of NNI-funded students who are hired for nanotechnology-related jobs. Number of published patents and applications (as reported by the U.S. Patent and Trademark Office) and patent licensing categorized according to —Inventor affiliation (academe, industry, government, individual). —Subject or sector (electronics, chemicals and materials, and so on). —Inventor’s country of origin. Number of Small Business Innovation Research (SBIR) awards related to nanotechnology, categorized by field of interest or topic. Number of nanotechnology-related companies partnering in specific ways with NNI-funded user centers, possibly weighted by funding levels. Number and economic health of companies started by NNI-funded SBIR and Small Business Technology Transfer (STTR) recipients. Nanotechnology-enabled products known to have been derived at least in part from NNI- funded activities. Progress in fostering the transfer of technologies into products for commercial and public benefit is difficult to define, assess, and quantify throughout the NNI given the complexity of interactions. The translation of NNI research into products will require different metrics for different agencies because the products will differ considerably in their type and path to fruition. Translational entities and programs set up by such agencies as DOD, the Department of Energy (DOE), NIST, and NSF may be dedicated to nano-enabled products or have goals that include nano-enabled products. Products vary considerably; for example, the products of NSF-funded university research are typically graduates, publications, and, to a smaller extent, intellectual property, all of which contribute to the development of the nanotechnology workforce and to the body of knowledge. DOD research is generally aimed at developing technology that can be deployed for the national defense. Many companies are interested in products and services for public sale. Standards developed by various standards-development organizations with the participation of NIST and other federal agencies are a public good that supports industry while reducing technical barriers that favor a particular company’s or country’s agenda. The pathways by which research results are translated into practical applications and commercial products are complex and numerous. Moreover, the time from research to product is typically measured in years or even decades. The NNI has existed for 10 years; nanotechnology-based products are emerging, and many more useful discoveries are in the innovation pipeline. At the agency or industry level, 18

OCR for page 16
mechanisms exist for technology transfer and commercialization, and different metrics may be required to capture their effectiveness. Moreover, commercialization depends on various innovation activities, and hence various metrics, in the NNI: knowledge generation and dissemination, technology transfer, commercialization, and workforce creation in which NNI agencies and program managers and members of the international nanotechnology R&D community are prime actors. Metrics may be based, for example, on knowledge (publications, intellectual property, and citations), workforce training (graduates, employees, and meetings attended), private-sector engagement (patent licensing data, SBIR or STTR grant data and later venture funding acquisition, cooperative R&D agreements, and public-private partnerships), or revenue. Desired outputs depend strongly on the agency involved; 26 agencies have widely different levels of engagement in the NNI as measured by funding for the research or staff involved. Outputs may even vary within a single agency. In DOE, for example, NNI-related output includes user centers, Advanced Research Projects Agency-Energy grants and contracts, SBIR funding, and the establishment of the Energy Frontier Research Centers program. In addition, outputs represent a broad range of technology readiness levels, and this has implications for the amount of funding, time, and effort required to convert a discovery or an invention into a useful product. Encouraging inventors to take risks to commercialize their ideas is as much a cultural issue as it is a financial or a technical issue. Commercialization can be stifled in an environment in which risk-taking is not encouraged, mentors are not available, or licensing is difficult; some regions and institutions are good at off entrepreneurial activities, and others are not. Those cultural issues are common to universities, government laboratories, and other research institutions and can create a bottleneck in the innovation pipeline. Although that is not a nanotechnology-specific problem, addressing it is important for removing barriers to commercialization of results, given the substantial investment in the NNI. Inventors and organizations may not be aware of the potential commercial value of technology if there is not an environment that encourages startups or spinoffs, and they may need a mechanism like a “preseed” workshop or NSF I-Corps 2 to foster commercialization concepts. Federal and local agencies have recognized that—the NSF I-Corps is an example of what can be done at the federal level to encourage and stimulate growth. It works to connect NSF-funded scientific research to the technologic, entrepreneurial, and business communities. The I-Corps curriculum is built on an accelerated version of Stanford University’s Lean LaunchPad course and additional elements designed for I-Corps grantees. All I-Corps team members attend a kickoff workshop at Stanford University, the Georgia Institute of Technology, or the University of Michigan and then join a series of Web-based lectures and present their business pitches at a meeting of I-Corps grantees. Awards are for $50,000 with a duration of 6 months. Many other excellent programs of this type may be available throughout the United States, but there is no current way to know how many and where they are. A measure of success for the NNI might be to expedite and facilitate connections for inventors in the nanotechnology-products realm to help them to identify agencies—federal, state, regional, and local—that can support them. The committee will examine such issues in its final report. NATIONAL NANOTECHNOLOGY INITIATIVE GOAL 3: TO DEVELOP AND SUSTAIN EDUCATIONAL RESOURCES, A SKILLED WORKFORCE, AND THE SUPPORTING INFRASTRUCTURE AND TOOLS TO ADVANCE NANOTECHNOLOGY The 2011 NNI Strategic Plan notes that the development and sustainment of the infrastructural elements addressed by NNI Goal 3 are essential for delivering commercial and public benefit from NNI efforts. The Strategic Plan supplements Goal 3 with three objectives that are paraphrased here as workforce development, informal education activities, and physical infrastructure development. 2 See http://www.nsf.gov/news/special_reports/i-corps/. 19

OCR for page 16
Definitions of success that might be applied to NNI Goal 3 include the following: The supply matches the demand for U.S.-based skilled nanotechnology workers. 3 Public understanding of and interest in nanotechnology and how it may affect our lives are expanded. The amount and the type of infrastructure for nanotechnology advancement are appropriate, given the funding levels. Users’ technical needs are met through NNI user facilities. Rates of use of NNI infrastructure are high. Possible metrics of progress as defined above in achieving NNI Goal 3 are listed below. Evidence that U.S.-based skilled nanotechnology workers trained through the NNI are fully employed. Evidence that there is not unmet demand for skilled nanotechnology workers. Numbers of people beyond the NNI research community reached by specific agency-driven outreach activities, such as teacher-education activities and K-12 student activities. Mass-media stories about nanotechnology activities in or related to NNI participating agencies. Use of current infrastructure, according to numbers and types of users, and the outcomes of use of the infrastructure. Satisfaction among participants in user facilities, as established through surveys. Responsiveness to unmet needs for infrastructure signaled by unfulfilled requests for access to infrastructure. The committee is impressed by the number and nature of programs targeting the training of a skilled nanotechnology workforce in the NNI environment. It is in the nation’s interest that the supply of and demand for skilled workers be in balance. It is therefore desirable to collect reliable data on the supply of and demand for workers who have critical skills. Even the number of students who are receiving formal, career-oriented, “nanotechnology” education at various levels funded by NNI agencies is difficult to assess with the current system for collecting data from the agencies that participate in the NNI; only some agencies appear to collect such data, and the National Nanotechnology Coordination Office does not aggregate the data that are available as far as the committee can tell. The committee is considering ways in which data on the supply of workers at all levels of training and education might be aggregated and compared with indicators of the workforce demand for skilled nanotechnology workers as a function of time. At a minimum, the NNI-funded ecosystem should be graduating students at a rate sufficient to drive the nanotechnology innovation and commercialization process. Achieving that result, however, will require as a first step the collection and analysis of data. It may, however, be useful to collect and analyze the supply-side dataset. For NNI participating agencies, it may be possible to report where students work immediately after graduation. NNI-trained students moving to employment with U.S. firms, agencies or with institutions involved in nanotechnology could perhaps be fairly viewed as expanding the skilled nanotechnology workforce, whether or not job listings specify nanotechnology skills. It is difficult to estimate the size of the current nanotechnology workforce, but the related issue of workforce growth in this segment, as estimated from periodic review of U.S. job listings, might provide a useful metric. The committee notes with interest the data on nanotechnology job openings collated by 3 A “nanotechnology worker” is, for example, a scientist or an engineer (such as a materials scientist, chemist, or physicist) who is trained to work on processes in the 1- to 100-nm range. 20

OCR for page 16
Freeman and Shukla for 2008 directly from the on-line job board SimplyHired.com. 4 The data are broken down into 18 categories, some of which are nanotechnology-specific (for example, scientist and engineer) and some of which might be considered support roles (information technology, human resources, and administration). Taken together, however, the data indicate the health of the U.S. nanotechnology economy. If tracked over a longer period, they might be considered a proxy indicator of the growth of the U.S. nanotechnology economy through the demand for a skilled nanotechnology workforce. The committee notes that many of the job listings represent workforce churn—skilled people changing jobs— rather than new positions, so it is the time-based growth in the number of listings that is of primary interest for NNI metrics, given the assumption that the churn rate might be taken as a somewhat constant fraction, other factors being equal. The number of people receiving “nanotechnology” education at various levels through outreach and informal educational activities enabled by the NNI and the effectiveness of such activities will probably also be important to quantify. It will be difficult to measure efforts to expand public understanding of nanotechnology and all that it entails or to measure the effectiveness of such efforts. A possible metric is an estimate of the number of people reached by specific agency-driven outreach activities. The NNI has created a substantial infrastructure that includes everything from laboratory equipment that is used by a single principal investigator to major facilities that are open to qualified researchers. The latter category includes the DOE nanoscale science research centers, the NIST Center for Nanoscale Science and Technology, the National Institutes of Health (NIH)-Food and Drug Administration (FDA)-NIST Nanotechnology Characterization Laboratory, and NSF centers and networks, including the National Nanotechnology Infrastructure Network and the Network for Computational Nanotechnology. 5 The committee applauds the objective stated in the 2011 NNI Strategic Plan of taking an inventory of current infrastructure and estimating infrastructure needs out to 2020. The related issue of accessibility of that infrastructure should also be addressed. Metrics of progress toward that objective should track how useful the current infrastructure is (for example, on the basis of numbers and types of users, rates of use of key tools, and outcomes of using the infrastructure) and whether there are unmet infrastructure needs. The committee is also interested in metrics that indicate the relative success of different models for operating the existing nanotechnology facilities in supporting innovation, such as papers written by academic and industry partners and related patent activity. Such metrics might reveal which operating models are most effective and thus provide direction to the management teams in new and existing facilities that are seeking to maximize impact. Some such data are given in the 2011 report Assessment of Fifteen Nanotechnology Science and Engineering Centers’ (NSECs) Outcomes and Impacts: Their Contribution to NNI Objectives and Goals. 6 NATIONAL NANOTECHNOLOGY INITIATIVE GOAL 4: TO SUPPORT THE RESPONSIBLE DEVELOPMENT OF NANOTECHNOLOGY NNI Goal 4 attempts “to assure that nanotechnology-enabled products minimize adverse impacts and maximize benefits to humans and the environment.” The NNI role in supporting responsible development includes investing in research on potential risks to health or the environment from 4 R. Freeman and K. Shukla, Jobs in Nanotechnology—Creating a Measure of Job Growth, Science and Engineering Workforce Project Digest, National Bureau of Economic Research, June 2008. 5 Information about each can be found on the nano.gov Web site by clicking on “Collaborations and Funding” and “User Facilities.” 6 Available at http://www.nsf.gov/crssprgm/nano/reports/Assessment_2011+May+12+of+NSEC+by+GaTech_ FinalReport_56p_web.pdf. 21

OCR for page 16
nanomaterials and on societal aspects of the development of nanotechnology applications. Ensuring responsible development also entails communicating relevant information with various stakeholders, including business, international governance and other organizations, educators, and the public. It is notable that success in responsible nanotechnology development is considered necessary for the achievement of NNI Goals 1-3. Of the eight NNI program component areas, two in particular reflect the goals of responsible development of nanotechnology: Environmental Health and Safety (EHS), and Education and Societal Dimensions. The 2011 NNI Environmental, Health, and Safety Research Strategy 7 lays out the breadth and complexity of NNI Goal 4 and supplements it with a number of important, and in many cases concrete, objectives. In 2012, the funding for EHS is estimated to increase by about 20 percent over 2011 levels. The increase is in keeping with the perception that EHS will be critical for success in leveraging nanotechnology for societal benefit by identifying and addressing potential hazards of nanomaterials at an early stage. The primary agencies, by dollar value, that are supporting the EHS program component area are NSF, NIH, the Environmental Protection Agency, and the National Institute for Occupational Safety and Health, and FDA is playing an increasing role as new nanotechnology products come to market. Although the Consumer Product Safety Commission has been a member of the NNI since 2004, it contributed to the NNI budget for the first time in 2011; this shows the increasing importance of Goal 4 as nanotechnology matures. Because of the complexity of NNI Goal 4, related definitions of success are particularly challenging to distill but may include the following: Development, updating, and implementation of a coordinated program of EHS research leads to development of tools and methods for risk characterization and risk assessment in general—including both hazards and the likelihood of exposure—and supports expanding understanding of potential risks posed by broad classes of nanomaterials. Results of EHS research worldwide are public and easily available to researchers and users of nanomaterials. Businesses of all sizes are aware of potential risks posed by nanomaterials and know where to obtain current information about their properties and best practices for handling them. To enable continued innovation, regulatory agencies have sufficient information to assess the risks posed by new nanomaterials. The NNI supports research to assess the societal effects of nanotechnology in parallel with technology development. K-12 students are exposed to nanotechnology as part of their education and are aware of the potential applications and opportunities available to those who go into STEM (science, technology, engineering, and mathematics) disciplines. The general public has access to information about nanotechnology and a growing percentage is familiar with the fundamental concepts. The NNI includes R&D aimed at applying nanotechnology to solve societal challenges, such as affordable access to clean water, safe food, and medical care. Possible metrics of progress toward success as defined above in achieving NNI Goal 4 are listed below. EHS collaborations and projects or centers funded. Number of NNI EHS research results that are made easily accessible, for example, through an NNI-managed clearinghouse or in cooperation with international organizations. 7 See http://www.nano.gov/sites/default/files/pub_resource/nni_2011_ehs_research_strategy.pdf. Accessed September 27, 2012. 22

OCR for page 16
Guidance documents developed and made available to the public. Number of faculty and students supported for research in nanotechnology-related endeavors. Number of K-12 students and educators engaged by NNI-funded researchers, including DOE laboratory outreach and NSF-funded researchers, and the effects of such engagement. Evidence of public awareness and attitude regarding nanotechnology based on data on NNI- funded research. Availability of on-line information and news items related to nanotechnology. Evidence that NNI agencies are engaged in international forums discussing and developing standards, norms, and strategies for responsible development of nanotechnology. Number of NNI participating agency representatives at various international forums. Compilation of commercialized or commercializable technologies. Number of companies offering EHS, nanotoxicity, or nanotechnology safety services. Evolution of outcomes and impact of sustained funding in the EHS and societal dimensions of the NNI. Progress toward Goal 4 requires collection of data and development of methods to assess potential risks associated with engineered nanomaterials. Integral to that effort is the design of methods and protocols for assessing properties of nanomaterials and their biologic effects on the environment and on human health and the creation of guidance documents, standards, or other regulatory approaches. The amount of information that is needed to make informed decisions is large (and expensive to collect and catalog). The committee applauds the NNI for its renewed commitment to addressing these hard problems and plans in its final report to suggest metrics for gauging progress or success without imposing undue reporting burdens on the participating agencies. THE PATH FORWARD TO IMPROVED METRICS The committee believes in the value of metrics—why we have them, what we hope to accomplish by using them, and how we can tailor them to yield the information desired—but will not recommend measuring something simply because it can be measured. Metrics should make clear what the desired outcomes are. That is, there must be a model that relates what is measured to the desired outcome and an accurate system for doing the measuring. Having both constitutes having a metric. Without both, measurements will have little value for program management. The committee recognizes the difficulty of defining robust models and metrics for a field as diffuse as nanotechnology, for agencies as diverse as the 26 NNI participating agencies, and for goals as far-reaching as the four NNI goals. However, it emphasizes that any models and metrics applied must be rigorous and able to stand up fully to scientific scrutiny. If the data used are inaccurate or if the models linking data to desired outcomes have not been properly established, evaluation, rational decision-making, and allocation of resources become compromised. For example, the definitions by various stakeholders of what counts as nanotechnology are not consistent and make comparing or combining current analyses difficult or impossible. The committee observes that data gathered by different agencies cannot now be usefully compared. The measurement systems are not the same. The agencies use different metrics for their R&D programs that are based on a given agency, its mission, and its historical way of doing things. The NNI is being asked to establish definitions of success and associated metrics for fulfilling the overarching NNI goals while meeting the needs and supporting the missions of the NNI participating agencies. To achieve those objectives, there must be both a model (or a set of models) that relates what is measured to the planned NNI outcomes and an accurate measurement system that operates throughout the NNI agencies. With respect to NNI R&D, some outcomes can be measured now; others may be measurable soon with the use of new data-collection and data-mining capabilities. In sum, what is needed to assess the NNI’s 23

OCR for page 16
progress and success are accurate measurement systems and valid models. In general, computational and data capacities have outrun the accuracy of measurement systems and understanding of the phenomena that relate metrics to desired outcomes. The result may be exciting graphical representations whose meaning remains uncertain. A key part of the solution is to get scientists together and to work with the NNI community to develop models that can be tested to validate the measures on the ground. In other words, the NNI could benefit from investing in research on indicators and processes to support the development and effective use of metrics. The issue of metrics is not peculiar to the NNI. Other federal research programs and the international R&D community also are grappling with the issue of how to measure impact and return on investment. The committee views the present study as an opportunity to stimulate additional discussion on the question of metrics. It believes that metrics and models that relate metrics to outcomes of R&D can and should be developed for the NNI and other government programs. This interim report presents an overview of considerations related to the characteristics of good metrics. The committee’s final report will provide specific recommendations on the topic that are based on the concepts presented here. 24

OCR for page 16
Appendixes

OCR for page 16