2
Measuring Progress and Defining Success in the Context of Federal Research Initiatives
The National Nanotechnology Initiative (NNI) is a component of the overall federal investment in science and technology (S&T), and the NNI goals fit within the broad S&T goals and priorities of the federal government as a whole. To assess the NNI and progress toward its goals, it is important to understand the framework within which it operates and how the definitions of success of federal initiatives are established.
The federal research and development (R&D) enterprise comprises programs in individual agencies focused on specific needs and missions and activities, such as the NNI, that involve a number of agencies and address national goals and priorities. The highest-level goals of federal S&T investment appear in various budget documents, such as the June 2012 memorandum on S&T priorities for the FY 2014 budget from Jeffrey Zients, acting director of the Office of Management and Budget (OMB), and John Holdren, director of the Office of Science and Technology Policy (OSTP):
Scientific discovery, technological breakthroughs, and innovation are the primary engines for expanding the frontiers of human knowledge and are vital for responding to the challenges and opportunities of the 21st century. We look to scientific innovation to
• Promote sustainable economic growth and job creation,
• Improve the health of the population,
• Move toward a clean energy future,
• Address global climate change,
• Manage competing demands on environmental resources, and
• Ensure the security of the Nation.
That memorandum1 and its predecessors2 articulated the continuing role of S&T in providing benefit to the nation and identified specific outcomes that the U.S. government and taxpayers expect from federal investment in S&T (Box 2.1).
The priorities memorandums also identified S&T priorities that required investment by and cooperation among multiple federal agencies and departments for success. In responding to those priorities, the heads of the executive departments and agencies were instructed to
Balance priorities to ensure resources are adequately allocated for agency-specific, mission-driven research while focusing resources, where appropriate, on addressing the (following) multi-agency research activities that cannot be addressed effectively by a single agency.
The 2014 R&D priorities document3 lists nine multiagency R&D priorities: advanced manufacturing; clean energy; global climate change; R&D for informed policy making and management; information technology R&D; nanotechnology (the NNI); biologic innovation; science, technology, and mathematics education; and innovation and commercialization.
Nanotechnology has been highlighted as a multiagency R&D priority in the annual priorities memorandums almost since the NNI was created. The memorandums call for agencies to “strengthen interagency coordination,” to find “novel approaches to collaboration,” and to support “joint programs using shared resources.” Whereas the NNI is emphasized as a priority of the current and past administrations, it also has been described as “a governmental initiative, representing a priority area for investment and activity, but not a distinct funding program with separate budget authority or central management.” The direction from OSTP and OMB for the agencies to focus resources in the NNI “where appropriate” and the lack of a clear management or budget authority for the Nanoscale Science, Engineering, and Technology (NSET) Subcommittee of the National Science and Technology Council Committee on Technology has led to an agency-mission focus
1 Memorandum for the Heads of Executive Departments and Agencies, “Science and Technology Priorities for the FY 2014 Budget,” M-12-15, June 6, 2012, available at http://www.whitehouse.gov/sites/default/files/m-12-15.pdf, accessed November 13, 2012.
2 For a summary, see R.M. Jones, “Making hard choices: OMB and OSTP issue guidance to agencies on formulation of FY 2014 budget requests,” FYI: The AIP Bulletin of Science Policy News, Number 88, June 19, 2012, available at http://www.aip.org/fyi/2012/088.html, accessed November 13, 2012; and Memorandum for the Heads of Executive Departments and Agencies, “FY 2009 Administration Research and Development Budget Priorities,” M-07-22, August 14, 2007, available at http://m.whitehouse.gov/sites/default/files/omb/assets/omb/memoranda/fy2007/m07-22.pdf, accessed November 13, 2012.
3 Memorandum for the Heads of Executive Departments and Agencies, “Science and Technology Priorities for the FY 2014 Budget,” M-12-15, June 6, 2012, available at http://www.whitehouse.gov/sites/default/files/m-12-15.pdf, accessed November 13, 2012.
BOX 2.1
Nanotechnology-Specific Excerpts from Representative OSTP-OMB Priorities Memoranda
2005 Priorities Document1
Nanoscale R&D priority areas continue to include material science and research relevant to medical care and homeland security. Though research at the nanoscale offers natural bridges to interdisciplinary collaboration, especially at the intersection of the life and physical sciences, the administration encourages novel approaches to accelerating interdisciplinary and interagency collaborations. Activities such as joint programs utilizing shared resources, as well as support for interdisciplinary activities at centers and user facilities, are encouraged.
2009 Priorities Document2
Robust federal investment in the agency programs that make up the NNI will expedite realization of the potential of nanotechnology to address national priorities in areas such as energy, security, health care, and the environment and will maintain U.S. scientific and technological leadership in this field.
Agencies should strengthen interagency coordination and support research on potential risks to human health and the environment, consistent with the National Science and Technology Council’s 2006 report, Environmental, Health, and Safety Research Needs for Engineered Nanoscale Materials.3
More broadly, the NNI should support both basic and applied R&D in nanoscience, develop instrumentation and methods for nanoscale characterization and metrology, and disseminate new technical capabilities to help industry advance nanofabrication and nanomanufacturing. Nanoscale research offers a natural bridge to collaboration between the life and physical sciences; therefore, agencies are encouraged to use approaches that accelerate interdisciplinary and interagency collaboration. Agencies are encouraged to participate in activities such as joint programs utilizing shared resources or leveraging complementary assets, as well as support for interdisciplinary activities at centers and user facilities.
2014 Priorities Document4
Within the interagency NNI, agencies should give priority to implementation of the 2011 Environmental, Health, and Safety Research Strategy,5 presenting an approach to ensuring the safe, effective, and responsible development and use of nanotechnology; and support for the Nanotechnology signature initiatives, which spotlight topical areas that represent key opportunities and can be more rapidly advanced through focused interagency R&D efforts.
1 FY 2005 Interagency Research and Development Priorities, available at http://www.whitehouse.gov/sites/default/files/omb/assets/omb/memoranda/m03-15.pdf, accessed November 13, 2012.
2 Memorandum for the Heads of Executive Departments and Agencies, August 14, 2007, available at http://m.whitehouse.gov/sites/default/files/omb/assets/omb/memoranda/fy2007/m07-22.pdf.
3 National Science and Technology Council (NSTC), Environmental, Health, and Safety Research Needs for Engineered Nanoscale Materials, Nanoscale Science, Engineering, and Technology Subcommittee, Committee on Technology, Executive Office of the President, September 2006, http://www.whitehouse.gov/files/documents/ostp/NSTC%20Reports/NNI_EHS_research_needs%202006.pdf, accessed October 1, 2013.
4 Memorandum for the Heads of Executive Departments and Agencies, June 6, 2012, available at http://www.whitehouse.gov/sites/default/files/m-12-15.pdf, accessed November 13, 2012.
5 NSTC, Environmental, Health, and Safety Research Strategy, Subcommittee on Nanoscale Science, Engineering, and Technology, Committee on Technology, Executive Office of the President, October 2011, available at http://www.nano.gov/sites/default/files/pub_resource/nni_2011_ehs_research_strategy.pdf, accessed October 1, 2013.
within the NNI for many agencies. All the member agencies see nanotechnology as enabling and the NNI as an important means of nurturing nanotechnology in the agencies, throughout federal R&D, and in R&D throughout the nation, but the agencies have used the NNI and its interagency bodies (the NSET Subcommittee and the National Nanotechnology Coordination Office [NNCO]) primarily as a vehicle for information-sharing and coordination of nanotechnology R&D activities. Program coordination and joint programs with shared resources have been planned and implemented by agencies “as appropriate” only when they support the primary missions of the agencies involved.
The conflict between the guidance from OSTP and OMB to the agencies to strengthen collaboration to meet the NNI goals and the agencies’ interpretation of “as appropriate” is a continuing source of tension between the NNI and those who review it. Entities that support oversight activities, such as the President’s Council of Advisors on Science and Technology (PCAST), which currently serves as the National Nanotechnology Advisory Panel called for by law, the Government Accountability Office (GAO), and the NRC panels that have advised and reviewed the NNI, have called upon the NNI agencies, through the NSET Subcommittee and the NNCO, to create long-term collaborations with a shared vision, supported by joint planning, coordination, and management. This sentiment dates back to the 2002 report of the NRC entitled Small Wonders, Endless Frontiers, which made recommendations for the initial organization and management of the NNI. That report included recommendations for the NSET Subcommittee to increase multiagency investments in research, in particular at the intersection of nanoscale technology and biology. This report recognizes the challenges to interagency programming and therefore calls for a special fund for Presidential grants, under OSTP management, to support interagency research programs relevant to nanoscale science and technology. The expectation of interagency collaboration also is reflected, for example, in the 2012 PCAST evaluation of NNI strategic planning: “While the NSET Subcommittee in 2011 produced a ‘National Nanotechnology Initiative Strategic Plan,’ individual agency contributions lack the cohesion of an overarching framework, and there is no clear connection between the goals and objectives of the NNI strategic plan with those of individual agencies.” That observation led the PCAST to recommend, as a first step, clarifying how the NNI fits into agency priorities and programs: “NNCO in partnership with OSTP should work with the agencies to develop implementation plans for achieving the goals and objectives outlined in the 2011 NNI strategic plan.”
The conflict is also reflected in the 2012 GAO report Nanotechnology: Improved Performance Information Needed for Environmental, Health, and Safety Research, in which GAO evaluated NNI environmental, health, and safety (EHS) documents according to its six desirable characteristics for a national strategy, as seen in Table 2.1. The report stated:
TABLE 2.1 Summary of Desirable Characteristics of a National Strategy
Desirable Characteristic | Brief Description | |
Purpose, scope, and methods | Addresses why the strategy was produced, the scope of its coverage, and the process by which it was developed. | |
Problem definition and risk assessment | Addresses the particular national problems and threats at which the strategy is directed. | |
Goals, subordinate objectives, activities, and performance measures | Addresses what the strategy is trying to achieve, steps to achieve the results, and priorities, milestones, and performance measures for gauging results. | |
Resources, investments, and risk management | Addresses what the strategy will cost, the sources and types of resources and investments needed, and where resources and investments should be targeted by balancing risk reductions and costs. | |
Organizational roles, responsibilities, and coordination | Addresses who will be implementing the strategy, what their and others’ roles will be, and mechanisms for them to coordinate their efforts. | |
Integration and implementation | Addresses how a national strategy is related to other strategies’ goals, objectives, and activities and to subordinate levels of government and their plans for implementing the strategy. |
SOURCE: GAO, Report to the Chairman, Committee on Environment and Public Works, U.S. Senate, Nanotechnology Improved Performance Information Needed for Environmental, Health, and Safety Research, Appendix I, accessed August 8, 2012.
NNI strategy documents for EHS research issued by the NSTC address two and partially address the other four of the six desirable characteristics of national strategies identified by GAO that offer a management tool to help ensure accountability and more effective results. For example, the NNI strategy documents provide a clear statement of purpose, define key terms, and discuss the quality of currently available data, among others. However, they do not include performance information—such as performance measures, targets, and time frames for meeting those measures—that would allow stakeholders to evaluate progress towards the goals and research needs of the NNI. In addition, the documents do not include, or sufficiently describe, estimates of the costs and resources needed for the strategy. Without this information, it may be difficult for agencies and stakeholders to implement the strategy and report on progress toward achieving the research needs and assess if investments are commensurate with costs of the identified needs.4
The concerns raised by the GAO report with respect to NNI EHS documents apply in general to the initiative as a whole. When NNI agencies do not share long-term goals as evidenced by joint planning, resource allocation, coordination, and
4 Government Accountability Office, Nanotechnology: Improved Performance Information Needed for Environmental, Health, and Safety Research, GAO-12-427, 2012, p. 1, available at http://www.gao.gov/assets/600/591007.pdf, accessed August 8, 2012.
management to reach the goals, it becomes difficult at best to define meaningful “performance measures, targets, and time frames for meeting those measures” or to assess “if investments are commensurate with costs of the identified needs.” As noted in Table 2.1, performance measures (“metrics”) follow directly from defining “what the strategy is trying to achieve, steps to achieve the results, and priorities, milestones, and performance measurers for gauging results.” The question is then to what extent the NNI agencies and the interagency NSET Subcommittee should be expected to create, implement, and assess a formal, long-term, interagency national strategy for the NNI that is based on the GAO model.
The present committee’s interim report assessed the suitability of current procedures and criteria for determining progress toward NNI goals and laid out possible definitions of success and associated metrics for the NNI on the basis of its four stated goals. (The interim report is Appendix E.) The committee found that the current procedures and criteria that the NSET Subcommittee and OSTP use for assessing progress toward NNI goals are embodied in annual supplements to the President’s budgets and the NNI strategic plans. As noted above, the annual supplements describe current and proposed investments in nanotechnology by agency; quantify investment by agency, program component area, and Small Business Innovation Research and Small Business Technology Transfer programs; and provide examples of specific accomplishments by individual agencies and by groups with respect to the four broad NNI goals5 and the objectives in the strategic plan. The NNI strategic plan provides “the framework that underpins the nanotechnology work of the NNI member agencies,” describes the interests of the individual agencies in nanotechnology R&D and the NNI, and identifies objectives and actions to reach the four goals.6 The implicit criteria for progress toward the NNI goals that the NSET Subcommittee and OSTP formally use are thus based on the quality of the specific examples of research or activities that meet the four goals in the programs of the individual agencies or in collaboration with other groups and on whether objectives in the strategic plan have been met.
Finding: The committee found that although the four NNI goals establish the scope of the NNI and the annual supplements are useful for communicating the breadth of NNI activities in the agencies, they are insufficient for measuring progress and guiding the management and coordination of the NNI.
5 See Box S.1 in the Summary.
6 National Science and Technology Council (NSTC), National Nanotechnology Initiative Strategic Plan, Committee on Technology, Subcommittee on Nanoscale Science, Engineering, and Technology, February 2011, available at http://www.nano.gov/sites/default/files/pub_resource/2011_strategic_plan.pdf, accessed December 19, 2012, p. 23.
In its interim report, the committee stated the need for an explicit framework for managing and coordinating the NNI that links investment, outputs, and short-term outcomes to specific long-term goals and links outcomes in the individual agencies and of interagency collaborations. Such a framework would include essential performance-management concepts, such as having a vision, goals, clearly articulated desired outcomes (long-term, medium-term, and short-term), accurate data on the resources and funding available and how they are allocated, agreed-on milestones and timelines, and models that link all these with specific agreed-on performance measures (metrics) chosen to relate short-term and medium-term outcomes and outputs to specific longer-term goals and outcomes.
That stated need is in line with the PCAST and GAO findings and recommendations. As an interim step, PCAST recommended that all the individual NNI agencies develop NNI implementation plans and articulate how their plans are related to the NNI strategic plan. Some of the participating agencies, such as the National Science Foundation and the National Institutes of Health, already have publicly articulated strategic plans or implementation strategies for nanotechnology, but most do not; see Box 2.2. Without goals that are more specific than the four general NNI goals, a public commitment to specific outcomes, and a publicly articulated performance-management framework, it is not possible to measure NNI progress toward its four goals. The committee’s interim report also noted that the suggestion for a framework is especially timely in light of the rapid development of computational tools and methods for collecting and analyzing data that may be useful for tracking and measuring the progress of the NNI and guiding its management.
Recommendation 2-1: An overarching definition of success for the NNI as a federal initiative should be evidence that NNI agencies are establishing and implementing an effective, explicit framework for planning, managing, and coordinating publicly identified NNI interagency programs, such as the signature initiatives. Such a framework should be based on essential performance-management concepts, and plans for and progress toward specific outcomes should be reported annually in the NNI supplement to the President’s budget.
In the following chapter, Chapter 3, the committee lays out some of the key factors to consider when developing a framework for planning, management, and coordination of NNI programs:
• Who benefits, and how? The NNI stakeholders are discussed in light of how they benefit from the NNI and what their definitions of success would be.
• What financial resources are available within the NNI, and for what purpose? The funding for NNI, which is all provided by the agencies, is
BOX 2.2
Current Processes and Procedures for Evaluating Progress Toward NNI Goals
Each year in the annual NNI Supplements to the President’s budget, the NSET and the NNCO report lists (1) budget and expenditures data allocated to the program component areas (PCAs) from each agency’s budget, (2) highlights from each agency programs, and (3) areas in which multiple agencies or external organizations have been active. Progress of the NNI against its four stated goals is reported in largely anecdotal form and is generally agency-centric. Several agencies have reported examples of successful nanotechnology-relevant programs and projects; some provide numerical data, and some have presented short summaries without many details. Interagency activities are reported in the same manner. This clearly reflects the priorities of the NNI agencies: NNI agencies manage their overall portfolios to focus on their primary missions, with nanotechnology being secondary.
For example, according to agency websites, the mission of the Department of Energy is to “ensure America’s security and prosperity by addressing its energy, environmental and nuclear challenges through transformative science and technology solutions;”1 the mission of the Department of Agriculture is to “provide leadership on food, agriculture, natural resources, rural development, nutrition, and related issues based on sound public policy, the best available science, and efficient management”;2 and the mission of the Department of Defense is to “provide the military forces needed to deter war and to protect the security of our country.”3
Programs and individual projects are monitored and evaluated within each agency with respect to its agreed-on mission-based deliverables by using processes and metrics developed by the agencies. But the evaluations typically are program-specific and agency-specific, and the deliverables and outcomes are generally reported in forms that cannot be easily aggregated and analyzed for their nanotechnology-related content. The committee found that no method or system is common to the NNI agencies for measuring and tracking progress toward NNI goals. Broad generalizations can be made, but there is little granularity except for specific examples of successful projects, discoveries, and products related to the agencies’ missions, which are mapped onto the four goals.
1 U.S. Department of Energy, About Us, available at http://energy.gov/about-us, accessed October 1, 2013.
2 U.S. Department of Agriculture, Mission Statement, available at http://www.usda.gov/fundinglapse.htm?navid=MISSION_STATEMENT, accessed October 1, 2013.
3 U.S. Department of Defense, About the Department of Defense (DOD), available at http://www.defense.gov/about/, accessed October 1, 2013.
analyzed in terms of the primary goals of the funding and the level of interagency interaction.
• What subjects in the NNI could benefit most from the framework? The NNI signature initiatives and the NSET Subcommittee working groups are examined in some detail.