1

Introduction

Research and development (R&D) organizations are operated by government, business, academe, and independent institutes. The success of their parent organizations is closely tied to the success of these R&D organizations. In this report, organization refers to an organization that performs research and/or development activities (often a laboratory), and parent refers to the superordinate organization of which the R&D organization is a part. Where the organization under discussion is formally labeled a laboratory, it is referred to as such. The questions arise: How does one know whether the organization and its programs are achieving excellence in the best interests of its parent organization? Does it have an appropriate research staff, facilities, and equipment? Is it doing the right things at high levels of quality, relevance, and timeliness? Does it lead to successful new concepts, products, or processes that support its mission and the interests of its parent?

How does one assess a research organization to answer these questions? This report offers assessment guidelines to senior management of R&D organizations and of their parent organizations and other stakeholders. The report lists the major principles of assessment, noting that details will vary from one organization to another. The report provides sufficient information to inform the design of assessments, but it does not attempt to prescribe precisely how to perform assessments.

CALL FOR THE STUDY BY THE NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY

The National Institute of Standards and Technology (NIST), a major, multiprogram government organization, promotes U.S. innovation and industrial competitiveness by advancing measurement science, standards, and technology in ways that enhance economic security and improve the quality of life for all Americans. Over its history there has always been at NIST the belief that a strong peer assessment of its programs is a critical aspect of ensuring that NIST is effectively fulfilling its mission. This assessment has traditionally been provided by the National Research Council (NRC) of the National Academies. Since 1959, the NRC has provided assessments of the technical quality of the scientific and technical work performed at NIST (formerly the National Bureau of Standards) laboratories.

In October 1, 2010, NIST implemented its first major reorganization in over two decades. The primary objective of the reorganization was to sharpen the focus of NIST’s programs on their respective missions and to optimize their ability to deliver both the cutting-edge research and the related services critical to the U.S. economy. The most notable impact of the NIST reorganization is its effect on the NIST laboratory programs: The reorganization shifted the NIST laboratories from being activity or discipline-based organizations to being mission- and outcome-oriented organizations. The reorganization condensed the number of NIST laboratories and focused them along the distinct NIST mission lines of metrology, technology, and the provision of unique user facilities. This reorganization prompted the need for a reevaluation of the NIST



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 11
1 Introduction Research and development (R&D) organizations are operated by government, business, academe, and independent institutes. The success of their parent organizations is closely tied to the success of these R&D organizations. In this report, organization refers to an organization that performs research and/or development activities (often a laboratory), and parent refers to the superordinate organization of which the R&D organization is a part. Where the organization under discussion is formally labeled a laboratory, it is referred to as such. The questions arise: How does one know whether the organization and its programs are achieving excellence in the best interests of its parent organization? Does it have an appropriate research staff, facilities, and equipment? Is it doing the right things at high levels of quality, relevance, and timeliness? Does it lead to successful new concepts, products, or processes that support its mission and the interests of its parent? How does one assess a research organization to answer these questions? This report offers assessment guidelines to senior management of R&D organizations and of their parent organizations and other stakeholders. The report lists the major principles of assessment, noting that details will vary from one organization to another. The report provides sufficient information to inform the design of assessments, but it does not attempt to prescribe precisely how to perform assessments. CALL FOR THE STUDY BY THE NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY The National Institute of Standards and Technology (NIST), a major, multiprogram government organization, promotes U.S. innovation and industrial competitiveness by advancing measurement science, standards, and technology in ways that enhance economic security and improve the quality of life for all Americans. Over its history there has always been at NIST the belief that a strong peer assessment of its programs is a critical aspect of ensuring that NIST is effectively fulfilling its mission. This assessment has traditionally been provided by the National Research Council (NRC) of the National Academies. Since 1959, the NRC has provided assessments of the technical quality of the scientific and technical work performed at NIST (formerly the National Bureau of Standards) laboratories. In October 1, 2010, NIST implemented its first major reorganization in over two decades. The primary objective of the reorganization was to sharpen the focus of NIST's programs on their respective missions and to optimize their ability to deliver both the cutting-edge research and the related services critical to the U.S. economy. The most notable impact of the NIST reorganization is its effect on the NIST laboratory programs: The reorganization shifted the NIST laboratories from being activity or discipline-based organizations to being mission- and outcome- oriented organizations. The reorganization condensed the number of NIST laboratories and focused them along the distinct NIST mission lines of metrology, technology, and the provision of unique user facilities. This reorganization prompted the need for a reevaluation of the NIST 11

OCR for page 11
processes for organizational assessment, to ensure that NIST is positioned to have the benefit of relevant feedback relating to all of its programs and mission areas. NIST's Visiting Committee on Advanced Technology (VCAT), an external advisory board, highlighted in 2010 the need for NIST to reevaluate its assessment and review processes. The VCAT 2010 Annual Report contained the following recommendation relevant to program assessment:1 Following the reorganization of NIST into mission-focused laboratories, NIST should develop a comprehensive assessment program that includes effective peer review of scientific quality, [and] customer satisfaction for measurement services and for effectiveness in meeting the needs of the particular measurement program. STATEMENT OF TASK In 2011, NIST charged the National Research Council to conduct a study of best practices in assessing R&D organizations and to prepare a report providing NIST with information on peer-review and performance-evaluation systems. The NRC formed the Panel for Review of Best Practices in Assessment of Research and Development Organizations to carry out this study. The Panel was charged by the Director of NIST to consider means of assessing specific organizational aspects in a manner that satisfies the requirements of NIST to perform effective assessments but also identifies assessment methods that can be applied selectively to other R&D organizations. The organizational aspects are: Technical merit and quality of the science and engineering work, The adequacy of the resources available to support high-quality work, The effectiveness of the agency's delivery of the services and products required to fulfill its goals and mission and to address the needs of its customers, The degree to which the agency's current and planned R&D portfolio supports its mission, The elements of technical management that affect the quality of the work, The extent to which the agency is accomplishing the impact it intends, and The agency's flexibility to respond to changing economic, political, social, and technological contexts. In this report, an R&D organization is readily understood to consist of a group of scientists, engineers, and support staff, with appropriate facilities and equipment, working to accomplish some stated mission goal(s). An R&D organization could be the Department of Defense (DOD) laboratories system, or it could be a single laboratory, or even a directorate, division, or group within a laboratory. It could be the entire array of Department of Energy (DOE) National Laboratories or any one of these. It could be a laboratory within an industrial organization. It could be an organized research center at a university, or a research center involving collaboration among several universities and industries. Even more broadly, an R&D 1 Visiting Committee on Advanced Technology, 2010. 2010 Annual Report of the Visiting Committee on Advanced Technology of the National Institute of Standards and Technology. U.S. Department of Commerce, Washington, D.C., p. 7. 12

OCR for page 11
organization might include such organizations as the National Science Foundation or the DOE funding offices. THE PANEL'S STUDY PROCESS To address the study charge, the NRC appointed 12 expert volunteers to the panel. These volunteers, whose biographical sketches appear in Appendix A, represent expertise across research, development, and management activities in the following organizational sectors: the federal government, industry, academia, and national laboratories. Several members of the panel had previously engaged in activities as members of NRC committees that assessed R&D activities at NIST and at other federal agencies. The panel gathered inputs from relevant literature and from formal discussions at panel meetings with representatives of the following organizations: NIST, the Defense Laboratory Office, the Office of Science and Technology Policy, the Office of the Director of National Intelligence, and the IBM Corporation. On March 19, 2012, the panel steered a workshop at the Keck Center of the National Academies in Washington, D.C., at which representatives from the following organizations provided formal presentations on assessment practices: Johns Hopkins University Applied Physics Laboratory, the Association of Public and Land Grant Universities, former congressional staff, Sandia National Laboratories, The Dow Chemical Company, and Microsoft Research; a presentation was also made by the former Assistant Secretary of the Army for Research, Development, and Acquisition. The workshop also garnered inputs through small group discussions with approximately 100 representatives from organizations of the federal government, industry, academia, and national laboratories. A summary of the workshop was published in September 2012.2 The panel conducted four meetings at which it considered the literature reviewed, the presentations made and the discussions conducted, and information derived from the extensive personal experience and expertise of the panel members, and the panel drafted this report summarizing their findings. The report underwent rigorous review by a committee of experts appointed by the National Research Council. When that review committee expressed satisfaction with the responses of the Panel to their review comments, and when the report was subsequently approved by the National Research Council, it was made available to the public. ORGANIZATION OF THE REPORT This report is organized to reflect the elements in the panel's statement of task. The body of the report consists of the Summary and six chapters. This first chapter describes the formation and study process of the panel. Chapter 2 presents issues that formed the framework within which the panel examined assessment practices. Chapters 3, 4, and 5 present assessment considerations applicable to the three foci of assessment considered fundamental by the panel: management practices, the technical quality of the R&D, and the impact of the R&D. Chapter 6 presents a summary of guidelines for consideration during an assessment. The report also includes 12 appendixes. In addition to the biographical sketches of the panel members (Appendix A), they present the following: a summary of a workshop presentation highlighting the importance of assessing the alignment between an organization's vision and its 2 National Research Council, 2012. Best Practices in Assessment of Research and Development Organizations-- Summary of a Workshop. The National Academies Press, Washington, D.C. 13

OCR for page 11
people (Appendix B); a discussion of considerations pertaining to the validation of assessments (Appendix C); an example of the effective application of peer advice during the planning phase of a research project at the Army Research Laboratory (ARL; Appendix D); a discussion of relevant statutes and requirements documents that may be involved in some types of assessments (Appendix E); a description of the process applied by the ARL to assess the relevance of its R&D (Appendix F); a discussion of stakeholder relationships for laboratories at the DOD, DOE, and NIST (Appendix G); a list of questions pertaining to the assessment of leadership and management (Appendix H); examples of recent cross-organizational assessments (Appendix I); a summary of assessment processes at NIST, ARL, and Sandia National Laboratories (Appendix J); a summary of assessment processes at selected other government organizations involving peer review of technical quality (Appendix K); and a set of assessment metrics and criteria applied by NRC panels that review the ARL. 14