Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 11
1
Introduction
Research and development (R&D) organizations are operated by government, business,
academe, and independent institutes. The success of their parent organizations is closely tied to
the success of these R&D organizations. In this report, organization refers to an organization
that performs research and/or development activities (often a laboratory), and parent refers to the
superordinate organization of which the R&D organization is a part. Where the organization
under discussion is formally labeled a laboratory, it is referred to as such. The questions arise:
How does one know whether the organization and its programs are achieving excellence in the
best interests of its parent organization? Does it have an appropriate research staff, facilities, and
equipment? Is it doing the right things at high levels of quality, relevance, and timeliness? Does
it lead to successful new concepts, products, or processes that support its mission and the
interests of its parent?
How does one assess a research organization to answer these questions? This report
offers assessment guidelines to senior management of R&D organizations and of their parent
organizations and other stakeholders. The report lists the major principles of assessment, noting
that details will vary from one organization to another. The report provides sufficient information
to inform the design of assessments, but it does not attempt to prescribe precisely how to perform
assessments.
CALL FOR THE STUDY
BY THE NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY
The National Institute of Standards and Technology (NIST), a major, multiprogram
government organization, promotes U.S. innovation and industrial competitiveness by advancing
measurement science, standards, and technology in ways that enhance economic security and
improve the quality of life for all Americans. Over its history there has always been at NIST the
belief that a strong peer assessment of its programs is a critical aspect of ensuring that NIST is
effectively fulfilling its mission. This assessment has traditionally been provided by the
National Research Council (NRC) of the National Academies. Since 1959, the NRC has
provided assessments of the technical quality of the scientific and technical work performed at
NIST (formerly the National Bureau of Standards) laboratories.
In October 1, 2010, NIST implemented its first major reorganization in over two decades.
The primary objective of the reorganization was to sharpen the focus of NIST's programs on
their respective missions and to optimize their ability to deliver both the cutting-edge research
and the related services critical to the U.S. economy. The most notable impact of the NIST
reorganization is its effect on the NIST laboratory programs: The reorganization shifted the NIST
laboratories from being activity or discipline-based organizations to being mission- and outcome-
oriented organizations. The reorganization condensed the number of NIST laboratories and
focused them along the distinct NIST mission lines of metrology, technology, and the provision
of unique user facilities. This reorganization prompted the need for a reevaluation of the NIST
11
OCR for page 12
processes for organizational assessment, to ensure that NIST is positioned to have the benefit of
relevant feedback relating to all of its programs and mission areas.
NIST's Visiting Committee on Advanced Technology (VCAT), an external advisory
board, highlighted in 2010 the need for NIST to reevaluate its assessment and review processes.
The VCAT 2010 Annual Report contained the following recommendation relevant to program
assessment:1
Following the reorganization of NIST into mission-focused laboratories, NIST
should develop a comprehensive assessment program that includes effective peer
review of scientific quality, [and] customer satisfaction for measurement services
and for effectiveness in meeting the needs of the particular measurement program.
STATEMENT OF TASK
In 2011, NIST charged the National Research Council to conduct a study of best practices
in assessing R&D organizations and to prepare a report providing NIST with information on
peer-review and performance-evaluation systems. The NRC formed the Panel for Review of Best
Practices in Assessment of Research and Development Organizations to carry out this study.
The Panel was charged by the Director of NIST to consider means of assessing specific
organizational aspects in a manner that satisfies the requirements of NIST to perform effective
assessments but also identifies assessment methods that can be applied selectively to other R&D
organizations. The organizational aspects are:
Technical merit and quality of the science and engineering work,
The adequacy of the resources available to support high-quality work,
The effectiveness of the agency's delivery of the services and products required to
fulfill its goals and mission and to address the needs of its customers,
The degree to which the agency's current and planned R&D portfolio supports its
mission,
The elements of technical management that affect the quality of the work,
The extent to which the agency is accomplishing the impact it intends, and
The agency's flexibility to respond to changing economic, political, social, and
technological contexts.
In this report, an R&D organization is readily understood to consist of a group of
scientists, engineers, and support staff, with appropriate facilities and equipment, working to
accomplish some stated mission goal(s). An R&D organization could be the Department of
Defense (DOD) laboratories system, or it could be a single laboratory, or even a directorate,
division, or group within a laboratory. It could be the entire array of Department of Energy
(DOE) National Laboratories or any one of these. It could be a laboratory within an industrial
organization. It could be an organized research center at a university, or a research center
involving collaboration among several universities and industries. Even more broadly, an R&D
1
Visiting Committee on Advanced Technology, 2010. 2010 Annual Report of the Visiting Committee on Advanced
Technology of the National Institute of Standards and Technology. U.S. Department of Commerce, Washington,
D.C., p. 7.
12
OCR for page 13
organization might include such organizations as the National Science Foundation or the DOE
funding offices.
THE PANEL'S STUDY PROCESS
To address the study charge, the NRC appointed 12 expert volunteers to the panel. These
volunteers, whose biographical sketches appear in Appendix A, represent expertise across
research, development, and management activities in the following organizational sectors: the
federal government, industry, academia, and national laboratories. Several members of the panel
had previously engaged in activities as members of NRC committees that assessed R&D
activities at NIST and at other federal agencies.
The panel gathered inputs from relevant literature and from formal discussions at panel
meetings with representatives of the following organizations: NIST, the Defense Laboratory
Office, the Office of Science and Technology Policy, the Office of the Director of National
Intelligence, and the IBM Corporation. On March 19, 2012, the panel steered a workshop at the
Keck Center of the National Academies in Washington, D.C., at which representatives from the
following organizations provided formal presentations on assessment practices: Johns Hopkins
University Applied Physics Laboratory, the Association of Public and Land Grant Universities,
former congressional staff, Sandia National Laboratories, The Dow Chemical Company, and
Microsoft Research; a presentation was also made by the former Assistant Secretary of the Army
for Research, Development, and Acquisition. The workshop also garnered inputs through small
group discussions with approximately 100 representatives from organizations of the federal
government, industry, academia, and national laboratories. A summary of the workshop was
published in September 2012.2
The panel conducted four meetings at which it considered the literature reviewed, the
presentations made and the discussions conducted, and information derived from the extensive
personal experience and expertise of the panel members, and the panel drafted this report
summarizing their findings. The report underwent rigorous review by a committee of experts
appointed by the National Research Council. When that review committee expressed satisfaction
with the responses of the Panel to their review comments, and when the report was subsequently
approved by the National Research Council, it was made available to the public.
ORGANIZATION OF THE REPORT
This report is organized to reflect the elements in the panel's statement of task. The body
of the report consists of the Summary and six chapters. This first chapter describes the formation
and study process of the panel. Chapter 2 presents issues that formed the framework within
which the panel examined assessment practices. Chapters 3, 4, and 5 present assessment
considerations applicable to the three foci of assessment considered fundamental by the panel:
management practices, the technical quality of the R&D, and the impact of the R&D. Chapter 6
presents a summary of guidelines for consideration during an assessment.
The report also includes 12 appendixes. In addition to the biographical sketches of the
panel members (Appendix A), they present the following: a summary of a workshop presentation
highlighting the importance of assessing the alignment between an organization's vision and its
2
National Research Council, 2012. Best Practices in Assessment of Research and Development Organizations--
Summary of a Workshop. The National Academies Press, Washington, D.C.
13
OCR for page 14
people (Appendix B); a discussion of considerations pertaining to the validation of assessments
(Appendix C); an example of the effective application of peer advice during the planning phase
of a research project at the Army Research Laboratory (ARL; Appendix D); a discussion of
relevant statutes and requirements documents that may be involved in some types of assessments
(Appendix E); a description of the process applied by the ARL to assess the relevance of its
R&D (Appendix F); a discussion of stakeholder relationships for laboratories at the DOD, DOE,
and NIST (Appendix G); a list of questions pertaining to the assessment of leadership and
management (Appendix H); examples of recent cross-organizational assessments (Appendix I); a
summary of assessment processes at NIST, ARL, and Sandia National Laboratories (Appendix
J); a summary of assessment processes at selected other government organizations involving peer
review of technical quality (Appendix K); and a set of assessment metrics and criteria applied by
NRC panels that review the ARL.
14
